This document provides a summary of distributed tracing in event-driven architectures using OpenTelemetry and Kafka. It discusses what distributed tracing is, the components of a distributed tracing system, and how OpenTelemetry can be used to instrument Kafka clients, Kafka Streams, and Kafka Connect. Specifically, it describes how the OpenTelemetry Java agent can be used to automatically instrument Kafka clients and stateless Kafka Streams processing. For stateful Kafka Streams processing and Kafka Connect, it discusses challenges and potential solutions around issues like caching and state management.

1. A Practical Guide To End-to-End Tracing
In Event Driven Architectures

2. Roman - UK developer at PIE Labs
PIE Labs, Confluent
• What is Distributed Tracing ?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• Kafka Connect instrumentation
• Summary
Who we are, what we’ll talk about…

3. What is Distributed Tracing (DT)?
https://www.altexsoft.com/blog/shipment-tracking-integration-apis-edis-carriers-aggregators/
Systems get
complex…

4. Components of a DT system
• Instrumentation
• Collection
• Visualisation
https://blog.gurock.com/distributed-tracing
/

5. What makes up a trace?
https://docs.logz.io/user-guide/distributed-tracing/what-is-tracing

6. Context and Context?
• Local Trace Context

7. Context and Context?
• Remote Trace Context

8. Trace Context and it’s Propagation?

9. Why Distributed Tracing?
Adding context to the message and process flow.
• Dependency graph
• Record of Event flow
• Log correlation
• Contextual metrics
• Answer questions like:
“This result looks weird. Show me all the intermediate
states, so I can debug where the weirdness started…”

10. Section: What is OpenTelemetry?
• What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• Kafka Connect instrumentation

11. Overview of OpenTelemetry
1. Standardised, vendor-agnostic
2. High-quality, ubiquitous, and portable
3. Collection of tools, APIs, and SDKs
4. Instrument, generate
5. Collect, and export
6. To an observability back-end (not OT - e.g. Jaeger)
7. To help you analyze your software’s performance and behavior

12. Support for Kafka in OpenTelemetry
Kafka Clients:
• Javaagent
• Tracing Wrappers
• Tracing Interceptors
Kafka Streams:
• Javaagent
• Supply Kafka Clients with Tracing
OpenTelmetry instrumentation

13. Javaagent - Auto Instrumentation Agent
• Aspect Oriented approach
• Byte Buddy
• Muzzle
• Extension support
• Service Provider Interface (SPI) for tracer customization
• Installed at runtime through `-javaagent` Java option
OpenTelmetry instrumentation agent - how does it work?

14. Tracing Interceptors
• Standard Consumer / Producer Interceptor
implementations
• Installed through Interceptor configuration
Tracing Interceptors - how do they work?

15. Tracing Wrappers
• Standard Consumer / Producer Interface
implementations
• Installed through code
• Uses java.reflect.Proxy to intercept relevant method
calls and add tracing behaviour
Tracing Wrappers - how do they work?

16. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• Kafka Connect instrumentation
Section: Kafka Client instrumentation

17. Tracing Consumer Producer Applications
Application Flow
Trace

18. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

19. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

20. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

21. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

22. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

23. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

24. Comparison of instrumentation methods
Interceptor / Wrapper
Javaagent

25. Comparison of instrumentation methods - Producer
Interceptor Javaagent
Wrapper

26. Comparison of instrumentation methods - Consumer
Interceptor Javaagent
Wrapper

27. Comparison of instrumentation methods
Interceptor Wrapper Javaagent
Installation Config or Code Code Runtime
Producer Metadata Limited Full Full
Consumer Metadata Full Full Full
Consumer Local Context Not propagated Not propagated Propagated
Library Kafka-clients-2.6 Kafka-clients-2.6 Kafka-clients-0.11

28. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• Kafka Connect instrumentation
Section: Kafka Client instrumentation

29. Kafka Streams support
• Kafka Streams specific Process span implementation in Javaagent
• Stateless Kafka Streams processing pipelines
• Limitations:
• Non-javaagent implementation limitations
• Stateful operation support
• Caching in stateful operations

30. Kafka Streams support - Stateless
Javaagent
Interceptor / Wrapper

31. Kafka Streams support - Stateless
Javaagent
Interceptor / Wrapper

32. Kafka Streams support - Stateless
Interceptor / Wrapper

33. Kafka Streams support - Stateless
Interceptor / Wrapper
Javaagent

34. Kafka Streams - Stateful - No Caching

35. Kafka Streams - Stateful - No Caching - Aggregate Span

36. Kafka Streams - Stateful - No Caching - Aggregate Span

37. Kafka Streams - Stateful - No Caching - Aggregate Span

38. Kafka Streams - Stateful - No Caching - Aggregate Span

39. Kafka Streams - Stateful - No Caching - Wrapped Store

40. Kafka Streams - Stateful - No Caching - Wrapped Store

41. Kafka Streams - Stateful - No Caching - Wrapped Store

42. Kafka Streams - Stateful - Caching

43. Kafka Streams - Stateful - Caching - Aggregate

44. Kafka Streams - Stateful - Caching - Aggregate

45. Kafka Streams - Stateful - Caching - Aggregate

46. Kafka Streams support - summary
• Supported as is with Javaagent:
• Stateless
• Stateful - with limitations - single thread context, - no caching
• Wrapping State Store approach
• Inlining Span creation into Stateful operations
• Transformer hack - possible repartitioning

47. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• Kafka Connect instrumentation
Section: Kafka Client instrumentation

48. Kafka Connect support - Source Task

49. Kafka Connect support - Source Task

50. Kafka Connect support - Source Task

51. Kafka Connect support - Source Task

52. Kafka Connect support - Sink Task

53. Kafka Connect support - Sink Task

54. Kafka Connect support - Sink Task

55. Future plans & More Information
confluent.io/community/ask-the-community
rkolesnev@confluent.io
https://github.com/rkolesnev/kafka-opentelemetry