An update on the Pinterest distributed tracing stack

1. P I N T R A C E
D I S T R I B U T E D T R A C I N G @ P I N T E R E S T
S U M A N K A R U M U R I

2. A B O U T M E
• Passionate about distributed tracing, observability and
cloud infrastructure.
• Lead on Visibility/Storage team at Pinterest.
• Lead for Zipkin project at Twitter.
• Ex-(Twitter, Facebook, Amazon, Yahoo, Goldman
Sachs).

3. M O T I VAT I O N

4. SPEED
IMPROVES
ENGAGEMENT

5. M I C R O - S E R V I C E S B R O K E O U R T O O L S
HOW DID THIS REQUEST EXECUTE?

6. A G G R E G AT E E V E N T S P E R S E R V I C E
U N D E R S TA N D T R E N D S A N D A L E R T S
C H E A P
S E R V I C E L E V E L O V E R V I E W
N O P E R R E Q U E S T O V E R V I E W
M E T R I C S

7. R E C O R D D I S C R E T E E V E N T S
M A N U A L C O R R E L AT I O N
E X P E N S I V E
F L E X I B L E B U T V E RY B R I T T L E
L O G S

8. P R O J E C T P R E S T I G E
P I N P O I N T
M A N U A L T R A C I N G

9. R E C O R D E V E N T S I N A R E Q U E S T W I T H C A U S A L
O R D E R I N G
What is Distributed Tracing?

10. S T R U C T U R E D L O G G I N G O N S T E R O I D S
A N N O TAT I O N , S PA N , T R A C E
What is Distributed Tracing?

11. T R A C E R E Q U E S T S : R E C O R D E V E N T S I N
A R E Q U E S T W I T H C A U S A L O R D E R I N G .
A C R O S S M O B I L E C L I E N T S , B A C K E N D
S E R V I C E S A N D D ATA B A S E S
Z I P K I N B A S E D T R A C I N G S O L U T I O N
M O R E E X P E N S I V E
P I N T R A C E

12. B U I L D I N G P I N T R A C E : 5 C H A L L E N G E S

13. B U I L D I N S T R U M E N TAT I O N
C H A L L E N G E 1
HARD & TEDIOUS
O N E I N S T R U M E N TAT I O N S P E R ( L A N G U A G E ,
F R A M E W O R K , T H R E A D P O O L , P R O T O C O L )
C O M B I N AT I O N .
P Y T H O N T R A C E R ( O P E N T R A C I N G A P I )
F I N A G L E Z I P K I N T R A C E R
C U S T O M A N D R O I D A N D I O S T R A C E R .

14. S PA N R E P O R T A N D A G G R E G AT I O N
C H A L L E N G E 2
First company wide span aggregation pipeline.

15. D E P L O Y I N S T R U M E N TAT I O N
C H A L L E N G E 3
3 instrumentations.
100+ services
40 teams
Sampling <1% traffic

16. T R A C E P R O C E S S I N G A N D S T O R A G E
C H A L L E N G E 4
Open sourced our streaming pipeline:
github.com/openzipkin/zipkin-sparkstreaming

17. T R A C E V I S U A L I Z AT I O N
C H A L L E N G E 5
Pintrace architecture

18. T R A C E S A R E D ATA
Z I P K I N U I

19. A P P L I C AT I O N S O F T R A C E D ATA
U N D E R S TA N D , D E B U G A N D T U N E D I S T R I B U T E D S Y S T E M S .

20. I D E N T I F Y I N G S E R V I C E S I N T E R A C T I N G
W I T H A R E Q U E S T
U N D E R S TA N D R E Q U E S T T I M E L I N E

21. I D E N T I F Y I N G D U P L I C AT E
C O M P U TAT I O N
U N D E R S TA N D R E Q U E S T T I M E L I N E
5% latency (20ms improvement) while halving the load

22. W H I C H C L U S T E R S E R V E D T H I S
R E Q U E S T ?
D E B U G D I S T R I B U T E D R E Q U E S T E X E C U T I O N

23. I D E N T I F Y I N G S E R I A L E X E C U T I O N
T U N E D I S T R I B U T E D S Y S T E M
Step pattern in a trace signifies serial execution
Parallel get_many after the bug fix.

24. T R A C E S A N D M E T R I C S
R E D U C E T I M E T O T R I A G E

25. S E R V I C E D E P E N D E N C Y G R A P H
U N D E R S TA N D D E P E N D E N C Y A C R O S S S E R V I C E S

26. E N D T O E N D T R A C I N G
T R A C E A U S E R I N T E R A C T I O N T O B A C K E N D C A C H E R E Q U E S T

27. T R A C I N G J S C L I E N T
T R A C E B R O W S E R E V E N T S
Identified real user latency was worse than
our measured latency.

28. T R A C I N G T H R O U G H C D N
C D N T R A F F I C A N A LY S I S
Clients send trace headers
to CDN.
Ingest traced fastly/cedexis
logs as spans.
Analyze spans to
compare CDN latency vs
user perceived latency.

29. M O R E A P P L I C AT I O N S O F T R A C E D ATA
• Improve time to triage.
• Automated root cause analysis.
• Tracking down p99 latencies.
• Identify architectural optimizations.
• Latency pipeline.
• Inter AZ traffic analysis.

30. L E S S O N S L E A R N E D
• User awareness and education are very important to
make tracing successful.
• Begin with the end in mind.
• Trace most valuable paths in the application.
• Distributed tracing landscape is confusing.
• Quality of traces is more important than quantity.

31. T E A M
N A O M A N A B B A S
J O N PA R I S E
B R I A N PA N E
S A M M E D E R
J O E G O R D O N
V I S I B I L I T Y T E A M
S R E T E A M
W E B / I O S ? A N D R O I D

33. Q U E S T I O N S ?
https://tinyurl.com/pintrace-architecture
https://tinyurl.com/pintrace-applications
https://tinyurl.com/pintrace-analysis
skarumuri@pinterest.com
twitter: @mansu