Most of Airbnb's development happens in several large monolithic repositories. With thousands of changes merged every day there is a high probability that changes that pass verification independently fail to integrate with other changes on the mainline. Evergreen is a system which guarantees serializability of changes. It enforces that any mainline commit passes all automated checks like compilation, static analysis or tests as if they were merged sequentially without overlapping in time. This is made possible by utilizing optimistic concurrency controls and build system topology in order to parallelize the verification of merging changes. Evergreen uses an actor model built on top of Kafka Streams.

1. Evergreen
Building Airbnb's Merge
Queue With Kafka Streams
Janusz Kudełka Joel Snyder

2. - Graydon Hoare
The Not Rocket Science Rule Of
Software Engineering:
"Automatically maintain a repository of
code that always passes all the tests"

3. Story Time

5. Main

6. PR 1
Main
initial_branch

7. initial_branch
PR 1
Main

8. Main
PR 1

9. Main
PR 1 PR 7 PR 11 PR 17 PR 25
docs_change
PR 27
PR 25

10. Main
PR 2 PR 7 PR 11 PR 17 PR 25
docs_change
PR 27
PR 25

13. Missing
Argument!

14. Solution: Run CI on Mainline to
detect Merge Races

15. new_feature
add_tests
PR 17
PR 25
PR 11
PR 11
Main
PR 2 PR 7 PR 11

16. new_feature
add_tests
PR 17
PR 25
PR 11
PR 11
Main
PR 2 PR 7 PR 11

17. add_tests
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17

18. Main
PR 2 PR 7 PR 11 PR 17 PR 25

19. Main
PR 2 PR 7 PR 11 PR 17 PR 25

20. Main
PR 12301 PR 12302
PR 12300
PR 1
PR 12302
PR 12302
PR 12302
PR 12302
PR 12302
PR 12302
PR 12302
PR 12302
PR 12302

21. Solution: pre-merge checks

22. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17

23. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17

24. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17

25. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
pre_merge_checks_for_add_logging
PR 17
PR 11 PR 25

26. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
pre_merge_checks_for_add_logging
PR 17
PR 11 PR 25

27. Main
PR 2 PR 7 PR 11 PR 17 PR 25

28. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
pre_merge_checks_for_add_logging
PR 17
PR 11 PR 25

29. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17

30. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11

31. add_logging
PR 25
PR 11
refactor
PR 26
PR 11
pre_merge_checks_for_add_logging
PR 17
PR 11 PR 25
pre_merge_checks_for_refactor
PR 17
PR 11 PR 26

32. add_logging
PR 25
PR 11
refactor
PR 26
PR 11
pre_merge_checks_for_add_logging
PR 17
PR 11 PR 25
pre_merge_checks_for_refactor
PR 17
PR 11 PR 26

33. Main
PR 2 PR 7 PR 11 PR 17 PR 25 PR 26

34. Solution:
serialized merge queue

35. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11

36. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 25
PR 17 PR 26 virtual_main
Merge
Queue

37. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 25
PR 17 PR 26 virtual_main
Merge
Queue

38. Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 25
PR 17 PR 26 virtual_main
Merge
Queue
PR 25

39. Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 25
PR 17 PR 26 virtual_main
Merge
Queue
PR 25

40. Main
PR 2 PR 7 PR 11 PR 17 PR 25 PR 26

41. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 25
PR 17 PR 26 virtual_main
Merge
Queue

42. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 17 PR 26 virtual_main
Merge
Queue

43. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17
refactor
PR 26
PR 11
PR 17 PR 26 virtual_main
Merge
Queue

44. add_logging
PR 25
PR 11
Main
PR 2 PR 7 PR 11 PR 17 PR 26

45. Downsides of the
(serialized) merge queue

46. Most engineers don't work on
the same services

47. Build Graph
core_lib
listing_lib
user_lib
reviews_lib
login_lib

48. Build Graph
core_lib
listing_lib
user_lib
reviews_lib
login_lib
Change to
listing_lib affects
reviews_lib

49. Build Graph
core_lib
listing_lib
user_lib
reviews_lib
login_lib
Change to
core_lib
affects
Everything

50. Build Graph Analysis
core_lib
listing_lib
user_lib
reviews_lib
login_lib
core_lib
listing_lib
user_lib
reviews_lib
login_lib
PR 1 PR 2

51. Build Graph Analysis
core_lib
listing_lib
user_lib
reviews_lib
login_lib
core_lib
listing_lib
user_lib
reviews_lib
login_lib
PR 1 PR 2

52. new_feature
add_tests
PR 17
PR 25
PR 7
PR 7
Main
PR 2 PR 7 PR 11

53. new_feature
add_tests
PR 17
PR 25
PR 7
PR 7
Main
PR 2 PR 7 PR 11
virtual_main_a
PR 7
Merge DAG
PR 17
PR 25
virtual_main_b

54. new_feature
add_tests
PR 17
PR 25
PR 7
PR 7
Main
PR 2 PR 7 PR 11
virtual_main_a
PR 7
Merge DAG
PR 17
PR 25
virtual_main_b

55. add_tests
PR 25
PR 7
Main
PR 2 PR 7 PR 11
PR 7
Merge DAG PR 25 virtual_main_b
PR 17
new_feature
PR 17
PR 7

56. add_tests
PR 25
PR 7
Main
PR 2 PR 7 PR 11
PR 7
Merge DAG PR 25 virtual_main_b
PR 17
new_feature
PR 17
PR 7

57. Main
PR 2 PR 7 PR 11 PR 17
new_feature
PR 17
PR 7
PR 25

58. End of
Part I
Flakiness
Fast Failure
Source Control Scaling
Priorities
Extensions
Merge Locks

59. Evergreen

60. Enqueue a PR with a comment
The PR will start pre-merge CI
The PR will be merged if checks pass
The
Interface

61. The UI

62. The UI

63. Nuances
Build graph topology
Pre-merge testing time
Pre-merge failure rate
Change velocity

64. Architecture

65. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

66. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

67. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

68. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

69. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Actions
Mutations

70. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

71. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

72. GraphQL
API
Transitions
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

73. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

74. Kafka

75. Transitions GraphQL
API
Evergreen Architecture
Producer
Source
Control
Supervisor Observer
Worker
UI
Webhooks
Events Events/Actions
Mutations

76. Why
Kafka
(Streams)
Event-based/asynchronous system

77. Why
Kafka
(Streams)
Event-based/asynchronous system
Single dependency

78. Why
Kafka
(Streams)
Event-based/asynchronous system
Single dependency
Exactly once guarantees

79. Why
Kafka
(Streams)
Event-based/asynchronous system
Single dependency
Exactly once guarantees
Sharding

80. Why
Kafka
(Streams)
Event-based/asynchronous system
Single dependency
Exactly once guarantees
Sharding
NoSQL

81. It just works
Kafka
Streams
Experience

82. It just works
Single event processing in Kafka Streams
Kafka
Streams
Experience

83. It just works
Single event processing in Kafka Streams
No replay snapshotting
Kafka
Streams
Experience

84. It just works
Single event processing in Kafka Streams
No replay snapshotting
Rebalancing
Kafka
Streams
Experience

85. It just works
Single event processing in Kafka Streams
No replay snapshotting
Rebalancing
Flaky Tests
Kafka
Streams
Experience

86. It just works
Single event processing in Kafka Streams
No replay snapshotting
Rebalancing
Flaky Tests
Non-programmatic operational tooling
Kafka
Streams
Experience

87. It just works
Single event processing in Kafka Streams
No replay snapshotting
Rebalancing
Flaky Tests
Non-programmatic operational tooling
Size Limits
Kafka
Streams
Experience

88. It just works
Single event processing in Kafka Streams
No replay snapshotting
Rebalancing
Flaky Tests
Non-programmatic operational tooling
Size Limits
DSL & Interactive Queries
Kafka
Streams
Experience

89. Questions?
janusz.kudelka@airbnb.com
All trademarks are the properties of their respective owners. Any use of these are for identification purposes only and do not imply sponsorship or endorsement.
joel.snyder@airbnb.com