Distributed tracing is used to analyze performance and error cases in service oriented architectures. The Observability team at Airbnb recently created Upshot, a data pipeline that uses Flink to analyze over 40 million trace events per minute. Summaries of the resulting data are sent to Druid, Datadog, and other downstream datastores. This talk will focus on how we use Flink and how we analyzed and addressed scaling issues we encountered while building Upshot.

1. Upshot: Distributed Tracing
with Flink
Brian Wolfe / 2018-09-05 / Flink Forward 2018

2. Tracing with Flink
● Observability at Airbnb
● Trace reconstruction job
● Successes (and failures)

3. Tracing with Flink
● Observability at Airbnb
● Trace reconstruction job
● Successes (and failures)

5. Airbnb
5 M +
Listings worldwide
81,000
Cities with listings
191+
Countries with listings
300
million+
Total guest arrivals

6. Observability

7. Observability
Ensure that software engineers at
Airbnb have the monitoring and
introspection tools to successfully
operate and develop their services.

8. The Reality

9. Reality 1: many tools

10. Reality 1: many tools
?

11. Reality 1: many tools

12. Reality 2: data silos

13. Reality 3: choosing the right data is hard
Present
Enrich
DS
Validate

14. Reality 3: choosing the right data is hard
Present
Enrich
DS
Validate
Late error in request

15. Reality 3: choosing the right data is hard
Present
Enrich
DS
Validate
Late error in request
Give me ALL the data!!

16. Tracing with Flink
● Observability at Airbnb
● Trace reconstruction job
● Successes (and failures)

17. Upshot Pipeline

18. Flink Aggregation

19. Flink Aggregation : CollectTrace
After one event

20. Flink Aggregation : CollectTrace
After more events

21. Flink Aggregation : CollectTrace
After all events

22. Flink Aggregation : CollectTrace
After all events

23. Tracing with Flink
● Observability at Airbnb
● Trace reconstruction job
● Successes (and failures)

24. Success: it works
3 Million Events / s

25. Success: One Instrumentation, Several Uses
High dynamic range histograms in DruidIndividual request visualizations
ResponseTime(ms)
Quantilems
Servicecall

27. Success : Post-Sampling of Events

28. Success: testing by dual-reading

29. Problem: serialization

30. Problem: serialization performance
3x performance improvement with a custom serializer

31. Problem: serialization performance
3x performance improvement with a custom serializer
String deserialization, alloc, GC is still > 25% of CPU cost.

32. Problem: coupling between all operators

33. Problem: coupling between all operators

34. Problem: coupling between all operators

35. Problem: coupling between all operators

36. Problem: coupling between all operators

37. Problem: coupling between all operators

38. Problem: coupling between all operators

39. Problem: coupling between all operators

40. Problem: coupling between all operators

41. Problem: coupling between all operators
Better after Flink 1.5, but still a problem

42. Redesigning the data flow

43. Problem: Coupling Between All OperatorsUseFlinkpartitionerto
writetoKafka

44. Problem: Coupling Between All OperatorsUseFlinkpartitionerto
writetoKafka
Limit fan-out

45. What is next?

46. Next Steps
● Localize impact of degraded performance
● Load shed when necessary
● Add more links between observability tools

47. Observability
Team
Willie Yao Nelson Gauthier Jerry Chung
Rong Hu Chen Luo Sarah Wada
Harry Shoff Nathan Baxter Joseph Sofaer