Slides from my talk at the "DevOps and the search for the Holy Grail" meetup in Stockholm on May 28, 2105 A short introduction on application monitoring & metrics with Prometheus, the monitoring system and time-series database. The use of Prometheus is illustrated with examples in Docker and Java. We'll use Grafana as a metrics dashboard on top of Prometheus and create a simple integration with Slack for alarm triggering. Meetup: http://www.meetup.com/DevOps-Stockholm/events/222471316/ Recording: https://youtu.be/Z0LlilNpX1U

1. What is your
application doing
right now?
Matthias Grüter
@mattgruter

3. It will happen to you!

4. Nagios
Monit
Consul
Pingdom
Munin
Curl
InfluxDB
Bosun
Graphite
Prometheus
Dashing
JMX
Riemann
Librato
Codahale Metrics
Collectd
...

5. Nagios
Monit
Consul
Pingdom
Munin
Curl
InfluxDB
Bosun
Graphite
Prometheus
Dashing
JMX
Riemann
Librato
Codahale Metrics
Collectd
...

6. Enter Prometheus
promotheus.io
by SoundCloud

7. Client Libraries: Java, Ruby, Go, Python, Bash, Node…
Exporters: Docker, Mesos, AWS CloudWatch, JMX, StatsD
Out-of-the-Box: etcd, cAdvisor, Kubernetes
Pushgateway
Scrape

8. Query
http_requests_total
http_requests_total{env=”prod”}
http_requests_total{name=~”worker”}

10. job: {
name: "container-exporter"
scrape_interval: "5s"
target_group: {
target: "http://exporter:9104/metrics"
}
}
prometheus.conf

13. [...]
rule_file: "/etc/prometheus/prometheus.rules"
prometheus.conf
ALERT InstanceDown
IF up == 0
FOR 5m
WITH {}
SUMMARY "Instance {{$labels.instance}} down"
DESCRIPTION "{{$labels.instance}} has been down
for more than 5 minutes."
prometheus.rules

14. notification_config {
name: "alerts"
slack_config {
webhook_url: <URL>
send_resolved: true
}
}
aggregation_rule {
repeat_rate_seconds: 300
notification_config_name: "alert"
}
Prometheus Alert Manager

16. Qubit Coins

19. job: {
name: "balance"
scrape_interval: "5s"
target_group: {
labels: {
label: {
name: "zone"
value: "eu-central-1"
}
label: {
name: "env"
value: "prod"
}
}
target: "http://webapp:8080/metrics"
}
}
prometheus.conf

20. static final Gauge numberOfCoins = Gauge.build()
.name("coins")
.help("Number of coins").register();
// …
numberOfCoins.set(42);
Java
● Gauges
● Summary
● Counter
● Histogram