There are two primary approaches to scrape and collect metrics using Prometheus - using a centralized set of dedicated scrapers or decentralized scrapers that run as an agent. With centralized scraping, Prometheus is deployed as a central scraper to pull metrics from all discoverable endpoints and sometimes can be split across multiple centralized instances using a few different approaches. However, with a decentralized approach, Prometheus runs as an agent and in Kubernetes is deployed as a DaemonSet on each node in a cluster and only collects metrics from the node it runs on. Each model has pros and cons - especially when operating at large scale - which can make it difficult when deciding which model to use. In this session, the speaakers will provide an overview of Prometheus metrics collection at DoorDash, where having highly reliable resources, easy endpoint discovery, and real-time insights is critical. They will share insights and best practices into DoorDash’s decision to implement a decentralized model by offering pros and cons of each approach. Leave with a better understanding of the “right” model for your use case(s).

1. Centralized vs
Decentralized
Centralized vs Decentralized
Prometheus Scraping Architecture
with DoorDash
Eric D. Schabell, Chronosphere
Ales Koprivnikar, Chronosphere
PromCon EU 2022

2. Life of a data point…
2

3. Life of a data point…
3

4. Life of a data point…
4

5. Scraping architectures
5
Centralized

6. Scraping architectures
6
Centralized Decentralized

7. Hybrid scraping architecture?
7
Centralized Decentralized

8. You CAN have all the cookies!
8

9. ABOUT
9

10. Metrics collection architecture
10
K8S
Node
K8S
Chronocollector
(daemonset)
Microservice
(container)
Microservice
(container)
Microservice
(container)
Statsd
Exporter
Chronocollector
Custom Log
Sources
Custom Log
Sources
Non-K8S
workloads
Custom
Exporter
/
EC2
Process
Process
Chronocollector
(systemd service)
Discovery
file

11. Models
11
• DaemonSet
‒ For microservices
‒ Standard config, rewrite rules
‒ Standard labels attached to each metric
• Sidecar (k8s)
‒ Suitable for noisy neighbors (can be
allocated more resources)
‒ Custom config
• Unix service
‒ For services on standalone machines
‒ Use file based discovery
‒ Drift possible
• Deployment (k8s or any
containerized env)
‒ Mainly external facing scraping
(Outside DD infra)
‒ Cloudwatch or vendor metrics
‒ Highly customized config (TLS auth,
rewrite rules)

12. What’s really important
12
• Stability of resource usage of
the scraper
‒ Every change impacts the whole cluster
• Discovery
‒ Performance in a big cluster
• Enriching metadata on metrics
in real-time
‒ Standardization of some labels
• Rewrite rules
‒ Performance penalties under multiple
rules
• A lot of recording rules
‒ SLO tracking

13. Decentralized scraping architecture
13
Pros:
• No single point of failure
• Scalable
• Flexible
• Negligible performance penalty for
rewrite rules
‒ we have a lot of them
• Noisy neighbor issues limited to local
machines

14. Decentralized scraping architecture
14
Pros:
• No single point of failure
• Scalable
‒ on our side!!
• Flexible
• Negligible performance penalty for
rewrite rules
‒ we have a lot of them
• Noisy neighbor issues limited to local
machines
Cons:
• Heavy resource usage
‒ Due to daemonset approach
‒ Most of the time, scale resources to the max -> lot
of resource waste
• Every change is a daemonset deploy in
multiple clusters
• Hard to debug issues

15. Some final thoughts…
15
• Distributed architecture is a must after a certain scale
‒ Plays very nicely with k8s
• There will be many, many exceptions
‒ A lot of helm charts, systemd service config
• Very stable (but resource heavy)

16. Questions

17. Thank You