Presented by Tom Wilkie at PromCon 2017 With Speaker Notes: https://goo.gl/V9qGva At PromCon 2016, I presented "Project Frankenstein: A multitenant, horizontally scalable Prometheus as a service". It's now one year later, and lots has changed - not least the name! This talk will discuss what we've learnt running a Prometheus service for the past year, the architectural changes we made from the original design, and the improvements we've made to the Cortex user experience.

1. Cortex: Prometheus as a
Service, One Year On
Tom Wilkie, PromCon 2017
tom.wilkie@gmail.com
https://github.com/weaveworks/cortex

3. https://www.youtube.com/watch?v=3Tb4Wc0kfCM

4. Cortex: Prometheus as a Service
• Natively multi tenant; isolate different
customers in the same services.
• Different story around scaling & HA
• “Virtually infinite” retention and durability
• Opportunities for performance
enhancements
Cortex
YourYourYourYour
Your Jobs
Alertmanager
Grafana Prometheus HA

5. Frontend
Ditributor
DynamoDB Memcache
Consul
Ingester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
S3
Cortex Architecture

6. A Year’s Evolution

7. Problem #1: DynamoDB Write
Throughput

8. https://github.com/weaveworks/cortex/issues/254

9. Frontend
Ditributor
DynamoDB Memcache
Consul
Ingester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
S3Table Manager
Cortex Architecture

10. Problem #2: DynamoDB Write
Throughput, again

11. Original schema:
• Hash Key: <user ID>:<hour>:<metric name>
• Range Key: <label name>:<label value>:<chunk ID>
New schema:
• Hash Key: <user ID>:<day>:<metric name>:<label name>
• Range Key: <chunk ID>:<chunk end time>
https://github.com/weaveworks/cortex/pull/262

12. Problem #3: Queries of Death

13. Frontend
Ditributor Querier
Table Manager DynamoDB Memcache
Consul
Ingester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
S3
Cortex Architecture

14. Problem #3: Recording rules
and alerts

15. Frontend
Ditributor Querier
Table Manager DynamoDB Memcache
Consul
Ingester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
S3
Ruler
Cortex Architecture

16. Problem #4: Long tail

18. https://www.weave.works/blog/the-long-tail-tools-to-investigate-high-long-tail-latency/

19. Problem #5: Cost

20. S3 DynamoDB
IOP Cost
($/IOP)
5x10-6 2x10-7
Storage Cost
($/GB/Month)
0.023 0.250
https://github.com/weaveworks/cortex/issues/141

21. 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0.055 0.06
0.005
0.01
0.015
0.02
0.025
Object size (GB)
Cost ($)
S3
DynamoDB

22. Frontend
Ditributor Querier
Table Manager DynamoDB Memcache
Consul
RulerIngester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
Cortex Architecture

23. Problem #6: DynamoDB, again

24. Frontend
Ditributor Querier
Table Manager BigTable Memcache
Consul
RulerIngester
Write requests
Read requests
Control requests
Prometheus
Your Jobs
Cortex Architecture

25. DynamoDB BigTable
99th Percentile Write
Latency (ms)
70-100 50-150
99th Percentile Read
Latency (ms)
100-2500 ~250
LOC ~2000 ~400
DynamoDB numbers courtesy of Weaveworks

26. Closing thoughts

27. 1. DynamoDB Write Throughput
2. DynamoDB Write Throughput, again
3. Recording rules and alerts
4. Long tail
5. Cost
6. DynamoDB, again

28. Running for >12months
• Availability: querier unavailable for <12hrs ~99.9%
• Durability: lost <2 days of data >99.5%
• 99th percentile write performance ~60ms
• 99th percentile query performance <200ms

29. Future
• Direct chunk writes from Prometheus to Cortex Chunk Store
• Separate ingester index for better load balancing
• Use prometheus/tsdb for the ingesters
• Etcd & gossip for ring storage
• Chunks in Google Cloud Storage

30. One more thing…

31. I left Weaveworks at the begging of June to
focus on Prometheus & Cortex development.
Since then I’ve teamed up with David to
develop some ideas around Prometheus,
logging, and tracing.
We’re available for Prometheus hosting,
consulting, training and support.
email: hello@kausal.co

32. Metrics

33. Logs

34. Traces

35. Thank you!
Questions?