Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Druid for real time monitoring & analytics at eBay

Slides posted with permission from eBay's Monitoring Platform team. The original talk was given at the Druid meetup on 2018-03-13, and the abstract was:

"At eBay, we are using Druid for application monitoring and analytics in near real time. We process data coming from thousands of apps which generates more than 100’s of Billion events per day. Druid helped us to monitor critical events and health of our apps in near real time and scale for our needs. We will be presenting how we built and scale Druid at eBay for our use-case in detail. We will be sharing challenges, learnings, and our journey building on this platform at eBay."

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Druid for real time monitoring & analytics at eBay

  1. 1. Motivation ➢ Existing system is legacy product. ➢ Scaling challenges with architecture evolution. ➢ Cardinality explosion with Microservices. ➢ Missing features like percentiles, topN.
  2. 2. Data received from Apps Messaging Layer Druid Deployment Apps Event Publisher Upstream Processor Kafka Architecture
  3. 3. ➢ Built on Kubernetes ➢ Indexed at 1 min, 15 min, 1 hour. Reindex for 1 day ➢ Median Query response times < 1 second ➢ Smart granularity selection for different range queries ➢ HDFS for deep storage Druid Deep Dive
  4. 4. Resiliency Architecture ➢ Dual-cluster setup ➢ Smart query routing ➢ Druid monitoring through events ➢ Resilient infra ➢ Two replicas of data per DC
  5. 5. Scale 5000+ Apps 4M+ Events/Sec 20M+ Unique Dimension Values 250+ Nodes in Druid Cluster 100s B Events Processed/Day
  6. 6. ➢ Used by SREs to monitor application health ➢ Used by developer community to monitor their apps ➢ Anomaly detection ➢ Alerting Customers & Usage
  7. 7. ➢ Percentiles ➢ Self-serve Alerting Next steps