Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Monitoring using Open source technologies

347 Aufrufe

Veröffentlicht am

In this presentation, we have talked about building an end to end solution for monitoring using open source technologies such as Graphite, Grafana, Apache Spark, Apache Kafka, OpenTSDB, Prometheus, etc.

Veröffentlicht in: Daten & Analysen
  • Login to see the comments

Monitoring using Open source technologies

  1. 1. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.cmu@gmail.com GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg
  2. 2. Hi, I am Jack.
  3. 3. Requirements: • 50,000 unique metrics from one source • Data points every minute • Roughly about 72 million data points per day • Data retention 60 days • User friendly UI with possible customization
  4. 4. Mission accomplished! 1 metrics source 50,000 unique metrics 72 million data points per day
  5. 5. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 400,000 unique metrics • About 600 million data points per day Team 3 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 4 Requirements: • 800,000 unique metrics • About 5 billion data points per day And more………
  6. 6. Should he continue with Graphite? Should he ask to reduce metrics or datapoints? How to dynamically scale Graphite? Does Grafana support other datasources? OpenTSDB / InfluxDB / KairosDB / Prometheus? Support scaling Infrastructure to support variable load of metrics? Challenges: • Multiple teams • Millions of unique metrics • Above 10 billion data points a day • Process 3 million logs every minute and generate metrics • Reprocessing of metrics and logs if needed • Provide real time monitoring for all of the above using GRAFANA!
  7. 7. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day
  8. 8. POC works for: 1 metrics source 50,000 unique metrics 72 million data points per day Team 1 requirements: 1 metrics source 100,000 unique metrics 200 million data points per day
  9. 9. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day
  10. 10. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
  11. 11. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
  12. 12. Clustering Graphite CARBON RELAY CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB . . . GRAPHITE WEB GRAPHITE WEB LOAD BALANCER
  13. 13. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
  14. 14. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
  15. 15. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
  16. 16. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time
  17. 17. Team 3 requirements: Over 5000 log sources 3 million logs per minute
  18. 18. Alerting
  19. 19. Graphite Stats - Apps using a stats library written by Alexander Filipchik Custom metrics - From other sources
  20. 20. (Subject to effort and time)
  21. 21. Alerting
  22. 22. • More than 3 million unique metrics supported - creation and deletion happens all the time • More than 11 billion data points written per day - across all TSDBs • Processing about 40 billion events per day - logs and metrics events in near real time (within 30 seconds) • More than 3000 requests per minute to Grafana dashboards - around 7000 requests in during outages
  23. 23. Alerting
  24. 24. https://grafana.net/plugins
  25. 25. http://grafana.org/ http://docs.grafana.org/ https://github.com/grafana/grafana https://raintank.slack.com
  26. 26. • Move • Copy • Extract • Insert • Remove • Rows • Panels • Template varia • Dashboard tag
  27. 27. • Dashboards • Datasources • Orgs • Rows • Panels • Template variables • Dashboard tags Version Control
  28. 28. • Production • Staging • Testing • Development Grafana in multiple environments
  29. 29. • Last 24 hours • By a dashboard tag • Customized dashboa Generate GIFs of important dashbo
  30. 30. Generate GIFs of important dashbo
  31. 31. • Upload/Store/Download dashboards to/in/from AWS S3 respectively. • Search/Download community dashboards from Grafana.net External features
  32. 32. https://utkarshcmu.github.io/wizzy-site/ https://utkarshcmu.github.io/wizzy-site/home/ https://github.com/utkarshcmu/wizzy https://raintank.slack.com/messages/wizzy/
  33. 33. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.cmu@gmail.com GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg

×