Imagine a world where you can access metrics, events, traces, and logs in seconds without changing code. Even more, a world where you can run scripts to debug metrics as code. In this session, you will learn about eBPF, a powerful technology with origins in the Linux kernel that holds the potential to fundamentally change how Networking, Observability, and Security are delivered. We’ll see eBPF in action applied to the Kafka world: identify Kafka consumers, producers, and brokers, see how they interact with each other and how many resources they consume. We'll even learn how to measure consumer lag without external components. If you want to know what’s next in Kafka observability, this session is for you.

1. Monitoring Kafka Without
Instrumentation Using eBPF
Anton Rodriguez
@antonmry

2. Our experience with Kafka
125 PiB of data per
month
3 B data points per
minute
production cluster of
~275 brokers was
running at 20 GB/sec

3. Kafka & observability

4. Kafka ecosystem

5. Kafka rebalances

6. Kafka Consumer Lag
https://github.com/linkedin/Burrow https://github.com/lightbend/kafka-lag-exporter

7. eBPF
eBPF programs (user-defined, sandboxed bytecode executed by the kernel) allow
user-defined instrumentation on a live kernel image that can never crash, hang or
interfere with the kernel negatively

8. eBPF & BCC
https://iovisor.github.io/bcc/

9. bpftrace
kubectl-trace
https://github.com/iovisor/bpftrace https://github.com/iovisor/kubectl-trace

10. eBPF Exporter
https://github.com/cloudflare/ebpf_exporter

11. ● Open-source CNCF observability platform
○ Using eBPF
● Automatically traces network messages
○ Kafka, HTTP, MySQL, etc.
○ Always active
● No instrumentation
○ No code modifications
○ No redeployments
What’s Pixie?
A CNCF sandbox project

12. Pixie’s approach
● Pixie Edge Module (PEM)
deploys on every node
● Capture data with eBPF
● Process data in user-space
(protocol parsing)
● Store data into tables for
querying by user

13. How is the data traced?
Pixie traces network-related Linux syscalls with eBPF kprobes.
In the case of Kafka:
● Fetch and Produce messages
● JoinGroup and SyncGroup messages, etc.

14. Demo time!

15. Summary
● Kafka observability is challenging
● eBPF opens a new world of possibilities
● Pixie provides auto-instrumentation for Kubernetes applications
○ No code modification
○ No redeployment
○ Easy to use
○ Specific domain metrics
○ Low overhead

16. Thank you!
Questions?
@antonmry