In this presentation, we have talked about building an end to end solution for monitoring using open source technologies such as Graphite, Grafana, Apache Spark, Apache Kafka, OpenTSDB, Prometheus, etc.

2. Utkarsh
Bhatnagar
• Senior Software Engineer @ Sony Interactive Entertainment (PlayStation).
• An active contributor to Grafana.
• Project initiator for wizzy – a user friendly CLI tool for GRAFANA
GitHub - https://github.com/utkarshcmu
Email – utkarsh.cmu@gmail.com
GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg

5. Hi, I am
Jack.

6. Requirements:
• 50,000 unique metrics from one source
• Data points every minute
• Roughly about 72 million data points per day
• Data retention 60 days
• User friendly UI with possible customization

10. Mission accomplished!
1 metrics source
50,000 unique metrics
72 million data points per day

11. Team 1 Requirements:
• 100,000 unique metrics
• About 200 million data points per day
Team 2 Requirements:
• 400,000 unique metrics
• About 600 million data points per day
Team 3 Requirements:
• 500,000 unique metrics
• About 2 billion data points per day
Team 4 Requirements:
• 800,000 unique metrics
• About 5 billion data points per day
And more………

13. Should he continue with Graphite?
Should he ask to reduce metrics or datapoints?
How to dynamically scale Graphite?
Does Grafana support other datasources?
OpenTSDB / InfluxDB / KairosDB / Prometheus?
Support scaling Infrastructure to support variable load of metrics?
Challenges:
• Multiple teams
• Millions of unique metrics
• Above 10 billion data points a day
• Process 3 million logs every minute
and generate metrics
• Reprocessing of metrics and logs if
needed
• Provide real time monitoring for all
of the above using GRAFANA!

14. Team 1 Requirements:
• 100,000 unique metrics
• About 200 million data
points per day
Team 2 Requirements:
• 500,000 unique metrics
• About 2 billion data
points per day
Team 3 Requirements:
• 3 million logs a minute
• Generate metrics in real
time
And more………
Team 1 Requirements:
• 100,000 unique metrics
• About 200 million data
points per day

15. POC works for:
1 metrics source
50,000 unique metrics
72 million data points per day
Team 1 requirements:
1 metrics source
100,000 unique metrics
200 million data points per day

17. Team 1 Requirements:
• 100,000 unique metrics
• About 200 million data
points per day
Team 2 Requirements:
• 500,000 unique metrics
• About 2 billion data
points per day
Team 3 Requirements:
• 3 million logs a minute
• Generate metrics in real
time
And more………
Team 2 Requirements:
• 500,000 unique metrics
• About 2 billion data
points per day

18. Team 2 requirements:
1 metrics source
500,000 unique metrics
2 billion data points per day

20. Team 2 requirements:
1 metrics source
500,000 unique metrics
2 billion data points per day

22. Clustering Graphite
CARBON
RELAY
CARBON CACHE
+ WHISPER +
GRAPHITE WEB
CARBON CACHE
+ WHISPER +
GRAPHITE WEB
CARBON CACHE
+ WHISPER +
GRAPHITE WEB
. . .
GRAPHITE WEB GRAPHITE WEB
LOAD
BALANCER

23. Team 2 requirements:
1 metrics source
500,000 unique metrics
2 billion data points per day
CR
G G G. . .
GW GW
LB

25. Team 2 requirements:
1 metrics source
500,000 unique metrics
2 billion data points per day
CR
G G G. . .
GW GW
LB

27. Team 2 requirements:
1 metrics source
500,000 unique metrics
2 billion data points per day
CR
G G G. . .
GW GW
LB

29. Team 1 Requirements:
• 100,000 unique metrics
• About 200 million data
points per day
Team 2 Requirements:
• 500,000 unique metrics
• About 2 billion data
points per day
Team 3 Requirements:
• 3 million logs a minute
• Generate metrics in real
time
And more………
Team 3 Requirements:
• 3 million logs a minute
• Generate metrics in real
time

31. Team 3 requirements:
Over 5000 log sources
3 million logs per minute

34. Alerting

35. Graphite Stats
- Apps using a stats library written by
Alexander Filipchik
Custom metrics
- From other sources

40. (Subject to effort and time)

42. Alerting

43. • More than 3 million unique metrics supported
- creation and deletion happens all the time
• More than 11 billion data points written per day
- across all TSDBs
• Processing about 40 billion events per day
- logs and metrics events in near real time (within 30 seconds)
• More than 3000 requests per minute to Grafana dashboards
- around 7000 requests in during outages

44. Alerting

47. https://grafana.net/plugins

50. http://grafana.org/
http://docs.grafana.org/
https://github.com/grafana/grafana
https://raintank.slack.com

52. • Move
• Copy
• Extract
• Insert
• Remove
• Rows
• Panels
• Template varia
• Dashboard tag

53. • Dashboards
• Datasources
• Orgs
• Rows
• Panels
• Template variables
• Dashboard tags
Version Control

54. • Production
• Staging
• Testing
• Development
Grafana in multiple environments

55. • Last 24 hours
• By a dashboard tag
• Customized dashboa
Generate GIFs of important dashbo

56. Generate GIFs of important dashbo

57. • Upload/Store/Download
dashboards to/in/from AWS S3
respectively.
• Search/Download community
dashboards from Grafana.net
External features

59. https://utkarshcmu.github.io/wizzy-site/
https://utkarshcmu.github.io/wizzy-site/home/
https://github.com/utkarshcmu/wizzy
https://raintank.slack.com/messages/wizzy/

60. Utkarsh
Bhatnagar
• Senior Software Engineer @ Sony Interactive Entertainment (PlayStation).
• An active contributor to Grafana.
• Project initiator for wizzy – a user friendly CLI tool for GRAFANA
GitHub - https://github.com/utkarshcmu
Email – utkarsh.cmu@gmail.com
GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg