Build Scalable Newsfeeds & Activity Streams - https://getstream.io Distributed tracing solves problems of identifying performance bottlenecks in distributed system. This presentation describes how Stream uses OpenTracing and Jaeger for distributed tracing in our system.

1. Distributed tracing
with Jaeger and OpenTracing

2. Which requests are slow? Why?
Lookup in Kibana:
Lacks context, why was it slow?
Grafana/Graphite:
We know what operations were slow but can’t
correlate them to any particular request
• Unclear how to aggregate data
• Lack of transparency and extensibility

3. Problem definition

4. • Understanding individual request behaviour
- Look at a specific request, not just an aggregated view
- Search by request ID
• Identification of performance bottlenecks
- Given a particular request data identify which operation was
causing performance issues
• Structured data access
- View N slowest request
- Search for a request on a particular shard/host or from a
particular client

5. Distributed tracing to the rescue!

7. • Pass around unique ID (Trace ID)
from service entry point to the end of execution
• Build a DAG of related operations
• Contextualise metadata

10. Vendor-neutral open API standard for distributed
tracing
• Decoupling tracing backend from client
• Library/framework integrations
- Integration via middleware for cross-service
tracing for a variety of transports

11. Metadata
•Tags
•Baggage
•Events
•Logs

12. How are we using OpenTracing

13. Integration with StreamGoKit:
Timer metrics can create spans via
*WithTracing method versions
Integration with API and Proxy servers:
Incoming request starts span, extracts relevant
metadata (shard, api key, etc.)

14. Tracing span information passed in code via context.Context object
Cross-service trace linking (Trace ID passed in HTTP headers/gRPC metadata)
func myOperation(...) {
timer, subcontext := metrics.StartWithTracing(context, "<operation>")
defer timer.Stop()
// timed operation
subTimer, _ := metrics.StartWithTracing(subcontext, "<sub-operation>")
// timed sub-operation
subTimer.Stop()
// timed operation continues
}

15. Jaeger

16. • Distributed tracing system released as open source
by Uber Technologies
- v1.0 released Dec 2017
• Written in Go

17. Flexible pipeline
• Agent
- abstracts routing and discovery of
collectors away from the client
- allows for adaptive sampling rate to
be implemented
• Collector
• UI for querying

18. Jaeger issues
Authentication for agent collector communication
Authentication for collector storage UI
UI is barely usable

19. Our deployment now (WIP)
• Collectors collocated with agents on service
machines
• Proxy, API, DB servers partially instrumented
• ElasticSearch as storage