1.
distributed
tracing

2.
twitter zipkin
google dapper
x-trace
tracelytics
... more!

3.
motivation

4.
what is slow?

5.
what is slow?

6.
causal flow of control

7.
causal flow of control

8.
how to

9.
possible approaches

10.
possible approaches
• Unique identifier

11.
possible approaches
• Unique identifier
• propagate throughout

12.
possible approaches
• Unique identifier
• propagate throughout
• write instrumentation for various
transports

13.
possible approaches
• Unique identifier
• propagate throughout
• write instrumentation for various
transports

14.
possible approaches
• Unique identifier
• propagate throughout
• write instrumentation for various
transports
• Observe and correlate

15.
possible approaches
• Unique identifier
• propagate throughout
• write instrumentation for various
transports
• Observe and correlate
• always on the outside - black box

16.
possible approaches
• Unique identifier
• propagate throughout
• write instrumentation for various
transports
• Observe and correlate
• always on the outside - black box
• difficult to get threaded + evented
processes right

17.
1BD57B58AE7E315BBEA
B6795F0BDC198296357

18.
t = start
nginx
cache python
db internet
the java

19.
t = start
nginx
cache python
db internet
the java

20.
t = start
nginx
cache python
db internet
the java

21.
t = start
nginx
cache python
db internet
the java

22.
t = start
nginx
cache python
db internet
the java

23.
t = start
nginx
cache python
db internet
the java

24.
t = start
nginx
cache python
db internet
the java

25.
t = start
nginx
cache python
db internet
the java

26.
t = start
nginx
cache python
db internet
the java

27.
t = start
nginx
cache python
db internet
the java

28.
t = start
nginx
cache python
db internet
the java

29.
t = start
nginx
cache python
db internet
the java

30.
t = start
nginx
cache python
db internet
the java

31.
t = start
nginx
cache python
db internet
the java

32.
t = start
nginx
cache python
db internet
the java
t = end

33.
piggyback rides
• More Doable
• HTTP: x-headers
• Thrift: secret argument
• Internal RPC protocol: you’re the
boss
• Less Doable
• SQL: one way ticket, also you’re
not percona
• memcache: not extensible so not
backwards compatible

34.
t = start
nginx
cache python
db internet
the java
t = end

35.
timing and structure
• Timing
• distributed = clock skew
• Structure -- two approaches
• Encode in ID
• Encode in back-pointers

36.
encode in ID?
• nginx1
• nginx1python1
• nginx1python1cache1
• nginx1python1cache1python2
• nginx1python1cache1python2sql
1
• nginx1python1cache1python2sql
1python3
• ...

37.
encode in back-pointer?
nginx python cache python

38.
reporting

39.
reporting

40.
other things worth
figuring out
• sampling
• reporting
• aggregate analysis

41.
thanks!
tracelytics.com
@dankosaur
dan@tracelytics.com

42.
resources
• X-Trace: http://x-trace.net
• http://x-trace.net/pubs/xtr-nsdi07.pdf
• Google Dapper: http://research.google.com/
pubs/pub36356.html
• Twitter Zipkin: https://github.com/twitter/zipkin
• CMU PDL: www.pdl.cmu.edu
• StarDust: http://www.pdl.cmu.edu/PDL-FTP/
SelfStar/thereska_sigmetrics06.pdf
• Trace Diff: http://www.pdl.cmu.edu/PDL-FTP/
SelfStar/NSDI11.pdf