Serverless applications are becoming a more and more attractive architecture of deploying software, providing many benefits over more traditional architectures. Unfortunately, due to the locked-in nature of Serverless platforms, monitoring options outside of vendor provided solutions are scarce. But does that mean we are doomed to be subject to Vendor lock-in? In this talk, Colin will cover the landscape of existing open source solutions to monitoring serverless applications, as well as present Cloudflares solution to how they monitor their increasingly large portfolio of serverless applications.

1. Monitoring in a Serverless
World
Colin Douch
Cloudflare Observability
OSMC 2022 |
@sinkingpoint |
@colind@hachydermio

2. In the beginning, life was good
These SaaS providers decided to invent Serverless Computing one day, and now we
have to support the Observability infrastructure
OSMC 2022 | @sinkingpoint | @colind@hachydermio

3. Serverless At Cloudflare
We made a thing
And then we started
using it…
OSMC 2022 | @sinkingpoint | @colind@hachydermio

4. A Quick Introduction
Hi! I’m Colin
Observability Lead @ Cloudflare
Incorrectly thinks that COBOL is
a decent language
OSMC 2022 | @sinkingpoint | @colind@hachydermio
Not British

5. Metrics
Logs
Traces
OSMC 2022 | @sinkingpoint | @colind@hachydermio

6. Metrics!
OSMC 2022 | @sinkingpoint | @colind@hachydermio
Prometheus:
Innovative,
Useful

7. The downside of Prometheus
Prometheus:
Innovative,
Useful,
Opinionated
OSMC 2022 | @sinkingpoint | @colind@hachydermio

8. Your system can be discovered and scraped
You have to wait
for Prometheus to
talk to you
OSMC 2022 | @sinkingpoint | @colind@hachydermio

9. Your system can expose metrics over the network
Prometheus needs to be able
to pull your metrics
- Your metrics need to be
exposed over HTTP
(generally…)
- And it needs to be
reachable over the network
OSMC 2022 | @sinkingpoint | @colind@hachydermio

10. Your system can do its own aggregation
What if your
service only ever
processes one
request?
Credit: Wikipedia
OSMC 2022 | @sinkingpoint | @colind@hachydermio

11. # TYPE slide SUMMARY
OSMC 2022 | @sinkingpoint | @colind@hachydermio

12. We did a survey!
- What sort of metrics do our
teams actually use?
- Do teams actually use
aggregation?
- What sort of semantics do they
need?
OSMC 2022 | @sinkingpoint | @colind@hachydermio

13. What We Found: 😱
OSMC 2022 | @sinkingpoint | @colind@hachydermio

14. What we found: Counters
Numbers that sum
together
- Request counts
- Error counts
OSMC 2022 | @sinkingpoint | @colind@hachydermio

15. What we found: Gauges
Numbers that go
up and down
- Memory usage
- Request Times
OSMC 2022 | @sinkingpoint | @colind@hachydermio

16. What we found: Histograms / Summaries
Deceptively hard.
- What happens when
you need to merge
buckets?
- What happens when
you get a summary
OSMC 2022 | @sinkingpoint | @colind@hachydermio

17. What we found: Info metrics
Metrics that replace
the entire metric family
when a new label set
comes in
- Version metrics
- Enums
OSMC 2022 | @sinkingpoint | @colind@hachydermio

18. Some prior art
OSMC 2022 | @sinkingpoint | @colind@hachydermio

19. Push Gateway
https://github.com/prometheus/pushgateway
OSMC 2022 | @sinkingpoint | @colind@hachydermio
Usually, the only valid use case for the
Pushgateway is for capturing the outcome
of a service-level batch job.

20. Aggregation Gateway
https://github.com/weaveworks/prom-
aggregation-gateway
OSMC 2022 | @sinkingpoint | @colind@hachydermio

21. An Internal Bespoke thing
Let’s just stitch the
two things together
OSMC 2022 | @sinkingpoint | @colind@hachydermio

22. Something Event Based?
We could leave Text Exposition
behind entirely
Events2prom, Cleodora
OSMC 2022 | @sinkingpoint | @colind@hachydermio

23. What we came up with
OSMC 2022 | @sinkingpoint | @colind@hachydermio

24. The Gravel Gateway
https://github.com/sinkingpoint/gravel-
gateway
Credit: Wikipedia
OSMC 2022 | @sinkingpoint | @colind@hachydermio

25. Some functional requirements
1. Accept pushes, like the Push
Gateway
2. Aggregate like the Aggregation
Gateway
3. Aggregate with all our identified
semantics
OSMC 2022 | @sinkingpoint | @colind@hachydermio

26. “How do we
express all the
semantics we
need, in the limited
amount of space
we have”?
OSMC 2022 | @sinkingpoint | @colind@hachydermio
# TYPE go_threads gauge
go_threads 13
# TYPE build_info
build_info{go_version="1.18",version="2022.2.1"} 1

27. That’s easy: We Cheat
Let’s smuggle semantic information
in the labels!
🤫
OSMC 2022 | @sinkingpoint | @colind@hachydermio

28. Introducing the “Clear Mode Label”
wshim_build_info{go_version=”1.18”,clear_mode=”info”}
OSMC 2022 | @sinkingpoint | @colind@hachydermio

29. Summing
wshim_request_count{clear_mode=”sum”} 1
wshim_request_count{clear_mode=”sum”} 1
wshim_request_count 2
OSMC 2022 | @sinkingpoint | @colind@hachydermio

30. Info Metrics
wshim_info{version=”1.1”,go_version=”1.18”} 1
OSMC 2022 | @sinkingpoint | @colind@hachydermio
wshim_info{version=”1.0”,go_version=”1.17,clear_mode=”family”} 1
wshim_info{version=”1.1”,go_version=”1.18”,clear_mode=”family”} 1

31. Means/Medians
wshim_memory_usage_bytes{clear_mode=”mean5m”} 0
wshim_memory_usage_bytes{clear_mode=”mean5m”} 5
wshim_memory_usage_bytes{clear_mode=”mean5m”} 10
wshim_memory_usage_bytes 5
OSMC 2022 | @sinkingpoint | @colind@hachydermio

32. Compatibility with existing client libraries
OSMC 2022 | @sinkingpoint | @colind@hachydermio

33. How this looks in practice
const { Pushgateway, register } = require('prom-client')
const pushGw = new Pushgateway(
'https://gravel.sinkingpoint.com',
register
);
const counter = new client.Counter({name: 'requests'});
counter.inc();
const info = new client.Gauge({name: 'info'});
info.set(1, {'clear_mode': 'family', 'version': '1.0'});
pushGw.push({ jobName: 'test' }, (err, resp, body) => {});
OSMC 2022 | @sinkingpoint | @colind@hachydermio

34. Looking Forward to the Future
OSMC 2022 | @sinkingpoint | @colind@hachydermio

35. Where Gravel Gateway goes from here
Credit: Wikipedia
OSMC 2022 | @sinkingpoint | @colind@hachydermio
- More work on scaling
- More interesting
aggregations
- Rewriting pebbles

36. Open Metrics
OSMC 2022 | @sinkingpoint | @colind@hachydermio

37. Cleodora and the shift towards Open Telemetry
OSMC 2022 | @sinkingpoint | @colind@hachydermio

38. Thanks!
OSMC 2022 | @sinkingpoint | @colind@hachydermio
Slides: https://colindou.ch/slides
Code: https://github.com/sinkingpoint/gravel-gateway
Want to chat, or tell me how bad of a person I am?
@colind@hachyderm.io
iam@colindou.ch
In the hallway, after this
Slides ^
Code v