The slides for my Prometheus talk at InfraCoders Vienna Meetup on 11.09.2017

1. 1
Oliver Moser, September 2017
Monitoring Stuff with Prometheus

2. Who am I
• Working at A1 and TAG for quite a while
• some Big Data
• DevOps/SRE/Containers/Orchestration
2

3. Fundamentals
3

4. Why do I need monitoring at all?
• Know when stuff breaks (and act on it)
• Understand the performance characteristics of your applications
• To meet service level objectives
• To improve performance/reliability

5. Fundamentals: Push vs Pull
Monitoring Service
HTTP Server
Metrics Agent
success_event
error_event
...
Monitoring Service
HTTP Server
Metrics Agent
GET /metrics
req_total
req_latency

6. Whitebox Monitoring
shows internal service information
e.g. request_processing_time,
request_errors_total
Fundamentals: Blackbox vs. Whitebox Monitoring
Blackbox Monitoring
restricted to external service
behaviour
e.g. ping, http
Agent
HTTP Server
HTTP GET / 200 OK
Agent
HTTP Server
GET /metrics
errors_total
req_total
req_latency

7. Prometheus Project Overview
• OpenSource monitoring tool, originally built at Soundcloud
• Heavily inspired by Google’s Borgmon
• Written in Go (mainly)
• Very active community (150+ contributors for core Prometheus)
• Member of the Cloud Native Computing Foundation

8. Prometheus Architecture
source: https://prometheus.io/docs/introduction/overview/

9. Prometheus Server
source: https://prometheus.io/docs/introduction/overview/

10. Service Discovery
source: https://prometheus.io/docs/introduction/overview/

11. Service Discovery
• Could live without SD in static environments
• In dynamic environments (e.g. Kubernetes) you must use SD
• Pods, Services,... come and go à impossible to statically configure
API Server
Pod 1
Pod 2
Pod n
...

12. Jobs, Targets and Exporters
source: https://prometheus.io/docs/introduction/overview/

13. 13

14. Graphing and Visualization
source: https://prometheus.io/docs/introduction/overview/

15. 15

16. Alertmanager
source: https://prometheus.io/docs/introduction/overview/

17. Alertmanager
• Alerts are configured in Prometheus and once triggered forwarded to
Alertmanager
• Alertmanager does
• Notifications (SMS, Slack, Pagerduty, Email etc)
• Deduplication/Grouping
• Silencing
• Inhibition
Prometheus
Server Alertmanager
forward ‘Instance Down’

18. Datamodel
18

19. Timeseries
• Tracks values of a metric over time (timestamp t, value v)
• Timestamps increase (strictly) monotonically
• 𝑡" < 𝑡"$% ∀ n ∈ ℕ
• Values can both increase or decrease
(t1,v1) (t2,v2) (t3,v3)
time

20. Metric Types
• Only relevant for Client libs (only untyped timeseries on the server)
• Counters: Values always increase
• requests_total, errors_total
• Gauges: Values can increase and decrease
• users_online, memory_free_bytes
• Histogram: Puts your measurements in buckets
• requests_latency_seconds_bucket
• Summary: Calculates percentiles over sliding time window
• requests_latency_seconds_summary

21. Labels
• A list of key/value pairs (𝑘% = 𝑣%, 𝑘. = 𝑣., … , 𝑘" = 𝑣")
• Labels partition a metric into timeseries
• So for every possible label combo in a given metric there will be a
timeseries is created
req_total{job=”job1", ver=”0.1”}: 10 à timeseries_1
req_total{job=”job1", ver=”0.2”}: 3 à timeseries_2
req_total{job=”job2", ver=”0.1”}: 4 à timeseries_3
req_total{job=”job1", ver=”0.1”}: 12 à timeseries_1
req_total{job=”job1"}: 1 à timeseries_4

22. Overall Data Model
events_processed_total
{component="enricher-deployment",
instance="10.244.2.98:8080",
job=”geo-enricher",
namespace="geo-analytics",
version="1.9.4-r5” } : 131241535
labels
metric name
label key label value metric value

23. PromQL
• Query language to select and aggregate timeseries
• Queries evaluate to either
• Instant Vector: Multiple timeseries with the same timestamp
• requests_total{job=”prom-boot”}
• Range Vector: Multiple timeseries with a range of timestamps
• rate(requests_total{job=“prom-boot”}[5m])
• Scalar: A single value
• sum(requests_total)

24. Alerting Rules
ALERT ErrorRateHigh
IF rate(errors_total[5m]) > 10
LABELS {
service = ”service_1",
severity = "warning"
}
ANNOTATIONS {
summary = "A high number of errors in service”,
description = "{{ $value }} errors have been registered
within the last hour for {{ $labels.instance }}"
}

25. Demo Time
25

26. Thanks!
26