By now, Prometheus has become the defacto standard for monitoring containerised applications. However, when it comes to monitoring multiple Kubernetes clusters through a single plane of glass, additional tools are required. In this talk, I will show how to setup a production monitoring landscape based on Prometheus and Thanos, spanning several Kubernetes clusters. Focussing on examples and best practices, I will also elaborate on how to secure communication between the individual components.

2. Pascal Fries
●
Kubernetes
●
Infra as Code
●
“cloud stuff”

3. who is using
Kubernetes?

4. who is using
Kubernetes?
single cluster?
multi cluster?

5. who is using
Kubernetes?
single cluster?
multi cluster?

6. who is using
Prometheus?

7. who is using
Prometheus?
standalone?

8. ●
service discovery
●
prom operator
●
kube prom stack
♥

9. multi cluster?

10. multi cluster? sure!
♥
♥
♥

11. and for a single
common endpoint...

12. ♥ ♥ ♥
●
remote write
●
remote read
●
federation

13. Awesome!

14. Awesome!
but...

15. ●
long term storage?
●
redundancy?
●
high availability?
●
multi tenancy?

16. how are you using
Kubernetes?
a customer’s
story

17. multiple
teams
●
3-4 dev
●
3-4 ops
●
1 infra

18. dev A
ops A
infra
ops A
dev B
infra
●
20+ clusters
●
shared
●
managed
●
owned

19. dev A
ops A
infra dev B
infra
requirement:
Single endpoint where
everyone can get their
metrics.

20. dev A
ops A
infra dev B
infra
other requirements:
●
long term storage
●
high availability
●
push based

21. ♥ ♥ ♥
; (

22. hanos
storage layer for
prometheus
querier | sidecar | store | receiver

23. hanos
●
truly cloud native
●
great for tinkering
●
remarkably resilient

25. hanos
how to set it up:
sample configurations

26. gRPC and TLS auth
instead of
REST and basic auth

27. ♥ ♥ ♥
querier
sidecar sidecar sidecar

28. long term
storage

29. ♥ ♥ ♥
querier
sidecar sidecar sidecar
S3
store

30. push based

31. ♥ ♥ ♥
querier
S3
store
receiver
remote write
compactor

32. microservices
♥

33. microservices
♥
https://www.bizpacreview.com/wp-content/uploads/2021/01/GettyImages-1206292055-1200x630.jpg

34. authenticating
connections
●
Thanos / Thanos: TLS auth
●
Prom / Thanos: basic auth

35. sharding &
redundancy
●
receiver as short term db
●
shard/replicate timeseries by labels
●
end up with duplicates in S3 :(

36. deduplication,
compaction, &
downsampling
●
querier deduplicates on the fly
●
compactor deduplicates in S3,
downsamples for faster queries

37. high availability
querier: stateless, replicated
receiver: stateful, replicated
compactor: cron, non-replicated

38. multi-tenancy:
a minimal model
●
compactor sets “tenant” label on
ingestion via relabeling config
●
querier augmented by
prom label proxy forces “tenant”
on queries

39. ♥ ♥ ♥
querier
S3
store
receiver
remote write
compactor

40. querier receiver
prom label proxy
auth proxy
- source_labels:
- cluster
- namespace
separator: "/"
regex: "dev/(appx|appy)"
replacement: "dev-b"
target_label: "tenant"
action: "replace"

41. relabeling
●
automate generation of config
●
use to force labels on
untrusted data
●
currently only reread on
receiver restart :(
#5398
- source_labels:
- cluster
- namespace
separator: "/"
regex: "dev/(ap
replacement: "d
target_label: "
action: "replac

42. summary
●
long term storage :)
●
redundancy :)
●
high availability :)
●
multi tenancy :|

43. extra stuff
●
Thanos ruler:
Prom rules for Thanos API
●
Thanos query frontend:
shard Thanos queries

44. Thank you!

45. questions?