Lightening talk at the first Devops Dublin meetup, giving an overview of Prometheus

1. Monitoring with Prometheus
(Lightning Talk)
Brian Brazil
Senior Software Engineer
Boxever

2. What is monitoring?
• Host-based checks

3. What is monitoring?
• Host-based checks
• High frequency information about a few key metrics

4. What is monitoring?
• Host-based checks
• High frequency information about a few key metrics
• High frequency high granularity profiling

5. What is monitoring?
• Host-based checks
• High frequency information about a few key metrics
• High frequency high granularity profiling
• Tailing logs

6. A different approach
What if we instrumented everything?
• RPCs
• Interfaces between subsystems
• Business logic
• Every time you’d log something

7. That’s a lot of metrics
That could be tens of thousands of codepoints across an
entire system

8. That’s a lot of metrics
That could be tens of thousands of codepoints across an
entire system
You’d need some way to make it easy to instrument all
code, not just the externally facing parts of applications

9. That’s a lot of metrics
That could be tens of thousands of codepoints across an
entire system
You’d need some way to make it easy to instrument all
code, not just the externally facing parts of applications
You’d need something able to handle a million time series

10. Presenting Prometheus
• Client libraries that make instrumentation easy
• Support for many languages: Python, Java, Go, Ruby…

11. Presenting Prometheus
• Client libraries that make instrumentation easy
• Support for many languages: Python, Java, Go, Ruby…
• Standalone server
• Can handle over a million time series in one instance
• No dependencies
• Easy to run

12. Presenting Prometheus
• Client libraries that make instrumentation easy
• Support for many languages: Python, Java, Go, Ruby…
• Standalone server
• Can handle over a million time series in one instance
• No dependencies
• Easy to run
• Integrations
• Machine, HAProxy, CloudWatch, Statsd, Collectd, JMX, Mesos,
Consul, Collectd, MySQL, cadvisor, etcd...

13. Presenting Prometheus
• Dashboards
• Promdash: Ruby on Rails web app
• Console templates: More power for those who like checking things in
• Expression browser: Ad-hoc queries
• JSON interface: Roll your own

14. Presenting Prometheus
• Dashboards
• Promdash: Ruby on Rails web app
• Console templates: More power for those who like checking things in
• Expression browser: Ad-hoc queries
• JSON interface: Roll your own
• Alerts
• Supports Pagerduty, Email, Pushover

15. But wait there’s more
Tired of aggregating and alerting off metrics like http.
responses.500.myserver.mydc.production?

16. But wait there’s more
Tired of aggregating and alerting off metrics like http.
responses.500.myserver.mydc.production?
Time series have structured key-value pairs, e.g.
http_responses_total{response_code=”500”,
instance=”myserver”,dc=”mydc”,env=”production”}

17. But wait there’s more
And you can do aggregation based on the key-value labels

18. But wait there’s more
And you can do aggregation based on the key-value labels
And do math

19. But wait there’s more
And you can do aggregation based on the key-value labels
And do math
Arbitrarily complex math

20. But wait there’s more
And you can do aggregation based on the key-value labels
And do math
Arbitrarily complex math
And all of this can be used in pre-computed rules and alerts

21. So, what does it cost?

22. So, what does it cost?
Less than $.001 per time series per month
All available under Apache 2.0 License
Developed by SoundCloud, Boxever and Docker
Many more contributors and users since launch

23. Too good to be true?

24. Too good to be true?
More powerful data model needs integrations and
instrumentation written to take advantage of that
Many features on roadmap:
• Service discovery
• Federation
• Long term storage
• HA Alertmanager
• More exporters, client libraries and integrations

25. Try it out!
http://www.boxever.com/tag/monitoring has step-by-step instructions on
monitoring:
• Machines
• Cassandra
• HAProxy
• Python Batch host-based jobs
• Java applications
Problems?
We’re on #prometheus on Freenode

26. More Information
http://prometheus.io
http://www.boxever.com/tag/monitoring
DevOps Ireland, April 2nd
Python Ireland, April 8th
SREcon15 Europe, May 14-15th