Modern systems today are far more complex to monitor. Microservices combined with containerized deployment results in highly dynamic systems with many moving parts across multiple layers. These systems emit massive amounts of highly dimensional telemetry data from hardware and the operating system, through Docker and Kubernetes, all the way to application and its databases, web proxies and other frameworks. Many have come to realize that the commonly prescribed Graphite+StatsD monitoring stack is no longer sufficient to cover their backs. New requirements need to be considered when choosing a monitoring solution for the job, including scalability, query flexibility and metrics collection. In this talk Horovits will look at the characteristics of modern systems and what to look for in a good monitoring system. He will also discuss the common open source tools, from the days of Graphite and StatsD to the currently dominant Prometheus. This talk will put you on the right track for choosing the right monitoring solution for your needs.

1. Monitoring Microservices
the Right Way
Dotan Horovits
Developer Advocate
@horovits

2. 2
• Developer Advocate at Logz.io
• 20 years in the hi-tech industry
• Developer, architect, product
• CNCF advocate, meetup organizer
• Podcast: OpenObservability Talks
Dotan Horovits
@horovits

3. Cloud-Native
Observability
https://logz.io/stackconf-2021/
Logs Metrics Tracing

4. Once upon a time…
i.e a few years ago

5. prod.webserver.html.request.total.count
StatsD Graphite
Metrics
Aggregated metrics
Metrics
Metrics
App
App
App

6. New paradigm, New
challenges

7. 7
Microservices instead of a Monolith

8. Kubernetes Control Plane (Master)
Virtual Machine
Kubernetes Node
API Etcd Sched C-M C-C-M
Pod Pod
Container
App
Container
App
Container
App
Container
App
Kubelet k-proxy
Virtual Machine
Kubernetes Node
Pod Pod
Container
App
Container
App
Container
App
Container
App
Virtual Machine
Kubernetes Node
Pod Pod
Container
App
Container
App
Container
App
Container
App
Cloud Native Architecture and Kubernetes
Kubelet k-proxy Kubelet k-proxy Kubelet k-proxy Kubelet k-proxy Kubelet k-proxy

9. Many Open Source & Cloud Services
to Monitor
Data Collection Data Analyzing Data Storing Data Querying
9

10. 10
Challenges
Microservices
Cloud Native
3rd Party OSS/Cloud
1.
2.
3.

11. New challenges call for
new capabilities

12. 12
Flexible Querying over High Cardinality
⇒
http_requests_total{env="prod",
service="nginx", server="pod7"}
prod.webserver.html.request.total
⇒
prod.node2.pod7.html.request.total
⇒
prod.nginx-service.html.request.total

13. Servers / VMs
Appliances / Infra
Services
/metrics
/metrics
/metrics
Metric Scraping and Autodiscovery

14. Servers / VMs
Appliances / Infra
Services
/metrics
/metrics
/metrics
Clustered Scalable Solution
Long-Term Storage
remote_write

15. SUMMARY

16. 16
1
Monitoring Microservices Requires
Flexible querying
with high cardinality
Efficient metric scraping
with autodiscovery
Scalability to handle
large volume of metrics
Open source to the rescue: Prometheus, OpenMetrics, TSDB, K8s/CNCF ecosystem
2 3

17. 17
For further information
● https://logz.io/blog/monitoring-microservices-best-practices/
● OpenObservability Talks podcast
● https://prometheus.io/
● https://openmetrics.io/
● Discussions on CNCF slack channels, gitter, mailing lists

18. Thanks
Dotan Horovits
Developer Advocate
@horovits