Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2DESAwC. Jessica Tai recaps her QCon SF 2018 “Great Migration” presentation then continues the story with a focus on how Airbnb is building, operating, and scaling its expanding network of services. Though their re-architecture to SOA is still ongoing, they are already seeing various benefits including improved performance, developer productivity, build and deploy times, and site reliability. Filmed at qconlondon.com. Jessica Tai has worked at Airbnb for 4 years, starting as a full-stack product engineer for the guest and host booking flow and is now an infrastructure engineer on the Core Services team. She leads the user data service, which is one of Airbnb’s highest QPS services and integrates with all business verticals. She is a member of Leadership & Development committee for women in tech at Airbnb.

1. Airbnb’sGreatMigration:
BuildingServicesatScale
JESSICA TAI / MARCH 4, 2019 / QCON LONDON

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
airbnb-services-scalability
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. @jessicamtai
2014

5. @jessicamtai
2015+

6. @jessicamtai
Hi,I’mJessica.
Ipairprogram
withmycorgi.

7. Why migrate? Service design tenets
Incremental
comparison
Best practices Results
Decomposition
@jessicamtai

8. Why migrate?
Best practices Results
Incremental
comparison
Service design tenets
Decomposition
@jessicamtai

9. Why migrate?
Best practices Results
Incremental
comparison
Service design tenets
Decomposition
@jessicamtai

10. Why migrate?
Best practices Results
Incremental
comparison
Service design tenets
Decomposition
@jessicamtai

11. Why migrate?
Best practices Results
Incremental
comparison
Service design tenets
Decomposition
@jessicamtai

12. Why migrate?
Best practices Results
Incremental
comparison
Service design tenets
Decomposition
@jessicamtai

13. Monorail,ourRubyonRailsmonolith
@jessicamtai

14. Easystartwithmonoliths
EARLY AIRBNB
Client traffic
Shared
database
Monorail
Data access query
Business logic
Presentation view
@jessicamtai

15. @jessicamtai

16. @jessicamtai

17. WHY DECIDE TO MIGRATE?
@jessicamtai

18. @jessicamtai

19. Moreincidents Slowerdeploytrains
@jessicamtai

20. Moreincidents Slowerdeploytrains
@jessicamtai

21. Oursolution:Service-orientedarchitecture(SOA)
NETWORK OF LOOSELY-COUPLED SERVICES
Client
API gateway
Service1
Service2
Database2Database3
Service3
@jessicamtai

22. CheckoutpageinSOA
Business travel service
Cancellation service
Home demand service
Pricing service
Home service
Reservation service
Messaging service
@jessicamtai

23. SERVICE DESIGN TENETS
@jessicamtai

24. Servicesownreads&writes
totheirdata @jessicamtai

25. Servicesaddressaspecificconcern
@jessicamtai

26. Avoidduplicatefunctionality
https://www.flickr.com/photos/popilop/331357312
@jessicamtai

27. Datamutationspropagate
viastandardevents @jessicamtai

28. Buildforproduction
@jessicamtai

29. DECOMPOSE BY
REQUEST LIFE CYCLE
@jessicamtai

30. Requestlifecycle
Client traffic
Shared
database
Monorail
Data access query
Business logic
Presentation view
V1: MONORAIL
@jessicamtai

31. Requestlifecycle
Monorail
API traffic
Routing & view
Business logic, model, data
via services
Client traffic
V2: MONORAIL & SERVICES
@jessicamtai

32. MIDDLE TIER
Shared business logic
Servicetypes
STRICT FLOW OF DEPENDENCIES
PRESENTATION
Synthesize
DERIVED DATA
Shared context, multiple sources
DATA
Entity read and writes
@jessicamtai

33. MIDDLE TIER
Shared business logic
PRESENTATION
Synthesize
DERIVED DATA
Shared context, multiple sources
DATA
Entity read and writes
Servicetypes
STRICT FLOW OF DEPENDENCIES
@jessicamtai

34. MIDDLE TIER
Shared business logic
PRESENTATION
Synthesize
DERIVED DATA
Shared context, multiple sources
DATA
Entity read and writes
Servicetypes
STRICT FLOW OF DEPENDENCIES
@jessicamtai

35. MIDDLE TIER
Shared business logic
DERIVED DATA
Shared context, multiple sources
DATA
Entity read and writes
PRESENTATION
Synthesize
Servicetypes
STRICT FLOW OF DEPENDENCIES
@jessicamtai

36. PRESENTATION
Synthesize
MIDDLE TIER
Shared business logic
DERIVED DATA
Shared context, multiple sources
DATA
Entity read and writes
Servicetypes
STRICT FLOW OF DEPENDENCIES
@jessicamtai

37. Homes
H O S T E D B Y K I T T Y · A P T O S , C A L I F O R N I A
Mushroom Dome
@jessicamtai

38. Homes data
service
Shared
database
Monorail
Data access query
Business logic
1.Migratingcoredatamodels
Presentation view
Homes
database
@jessicamtai

39. Homes data
service
Shared
database
Monorail
Data access query
Business logic
2.Migratingcorebusinesslogic
Presentation view
Homes
database
Pricing derived
data service
Pricing trends
data store
@jessicamtai

40. Homes data
service
Shared
database
Monorail
Data access query
Business logic
3.Migratingcoreproductviews
Presentation view
Homes
database
Pricing derived
data service
Pricing trends
data store
Checkout
presentation
service
@jessicamtai

41. Homes data
service
Shared
database
Monorail
Data access query
Business logic
4.Migratingcoreproductwrites
Presentation view
Homes
database
Homes
validation
middle-tier
Pricing derived
data service
Pricing trends
data store
Checkout
presentation
service
@jessicamtai

42. Requestlifecycle
Monorail
API traffic
Routing & view
Client traffic
V2: MONORAIL & SERVICES
@jessicamtai
Business logic, model, data
via services

43. Requestlifecycle
API gateway
Middleware
Session data
service
Authentication
data service
Oauth data
service
Risk derived
data service
...
Request
context
Presentation, logic, data
V3: SOA & API GATEWAY
Routing
@jessicamtai

44. Requestlifecycle
API gateway
Middleware
Session data
service
Authentication
data service
Oauth data
service
Risk derived
data service
...
Request
context
Web rendering
service
HTML viewV3: SOA & API GATEWAY
Routing
@jessicamtai
Presentation, logic, data

45. Monolith
world
Services
world
TheFutureTM
@jessicamtai

46. Migration
world
Monolith
world
Services
world
@jessicamtai

47. COMPARE FOR DIFFERENCES
@jessicamtai

48. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Read path A
Read path BMonorail
Service
Database
@jessicamtai

49. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

50. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

51. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

52. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Production
traffic
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

53. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
100%
Production
traffic
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

54. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
100%
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

55. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Monorail
Service
Database
Response A
Response B
Consumer +
offline
comparison
framework
@jessicamtai

56. Dualreadcomparison
WaitRamp&waitCompareGate
Admin web UI
configuration
1% traffic Gradual
increments
Gather traffic
patterns
All traffic through
service only
Switch
Monorail
Service
Database
The only read path
@jessicamtai

57. Writecomparison
DUAL WRITE TO SEPARATE DATABASES
Presentation service
Write validation
middle tier service
Write path A
Write path B
Data service
(production)
Data service
(shadow)
Consumer +
offline comparison
framework
Monorail
Payload A Payload B
@jessicamtai

58. s
Writecomparison
DUAL WRITE TO SEPARATE DATABASES
Presentation service
Write validation
middle tier service
The only write path
Data service
(shadow)
Monorail
@jessicamtai

59. APIgatewaycomparison
API gateway
Presentation service
Product logic
Monorail
Product logic
Original request
(no middleware applied)
Shadow request
with context
Add request context
@jessicamtai
Middleware
@jessicamtai
Shadow
request copy

60. APIgatewaycomparison
API gateway
Presentation service
Product logic
Request
with context
Middleware
@jessicamtai

61. Incrementalmigration
@jessicamtai

62. ! Production traffic with partially complete service
○ e.g. batch API /loadUsers
○ Fetch users only by user id
! Unblock clients
Migratebyendpoint
@jessicamtai

63. Migratebyattribute
Service Monorail
Database
Read migrated
attributes
Read not-yet-migrated
attributes
Database
Presentation service
Production traffic
@jessicamtai

64. SOA BEST PRACTICES
@jessicamtai

65. Frameworks
Auto-generate code
Testing&deploying
Replay production traffic
Observability
Standard templates
Standardizeservicebuilding
SCALE WITH CONSISTENCY
@jessicamtai

66. Service
Service&clientsetup
Business logic
@jessicamtai

67. Service
Service&clientsetup
Business logic
Endpoint logic
Server
transport
@jessicamtai

68. Service
Service&clientsetup
Business logic
Endpoint logic
Server
transport
Java client
Ruby client
Client
transport
Client
transport
@jessicamtai

69. Service
Service&clientsetup
Business logic
Server metrics
Server
diagnostics
Startup /
teardown
Endpoint logic
Metrics
Data
validation
Server
transport
Server
resilience
Java client
Ruby client
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Type
checking
@jessicamtai

70. Service
Endpoint logic
Service&clientsetup
Business logic
Server metrics
Server
diagnostics
Startup /
teardown
Dashboard
Dashboard
Alert
Alert
Alert
Runbook
documentation
Metrics
Data
validation
Server
transport
Server
resilience
Java client
Ruby client
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Type
checking
@jessicamtai

71. Service
Endpoint logic
Service&clientsetup
Business logic
Server metrics
Server
diagnostics
Startup /
teardown
Dashboard
Dashboard
Alert
Alert
Alert
Runbook
documentation
Metrics
Data
validation
Server
transport
Server
resilience
Java client
Ruby client
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Type
checking
@jessicamtai

72. FrameworksusingThriftIDL
@jessicamtai

73. IDL
Service
Endpoint logic
Business logic
Server metrics
Server
diagnostics
Startup /
teardown
Dashboard
Dashboard
Alert
Alert
Alert
Runbook
documentation
Metrics
Data
validation
Server
transport
Server
resilience
Java client
Ruby client
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Metrics
Client
transport
Data
validation
Error
handling
Resilience
Type
checking
@jessicamtai

74. ThriftIDL
API FRAMEWORK
/** Batch request for demo data */
struct LoadSomeDataRequest {
1: optional set<i64> ids
/** Some extra context baz */
2: optional bool fooBar (personal)
}
@jessicamtai

75. ThriftIDL
API FRAMEWORK
/** id to data response */
struct LoadSomeDataResponse {
1: optional map<i64, SomeData> data
}
@jessicamtai
/** Batch request for demo data */
struct LoadSomeDataRequest {
1: optional set<i64> ids (personal)
/** Some extra context baz */
2: optional bool fooBar
}

76. /** /loadSomeData batch endpoint */
LoadSomeDataResponse loadSomeData
(1: LoadSomeDataRequest request)
throws (1: SomeException exception1)
(accept_replay = "true", rate_limit = "true")
ThriftIDL
API FRAMEWORK
@jessicamtai

77. Thrift annotations
@jessicamtai

78. Block comments
from .thrift file
@jessicamtai

79. TRY TRY AGAIN SUCCESS
FAIL FAST
@jessicamtai

80. Separateasync
workerthreadpools
Graceful
degradation
@jessicamtai

81. Testing&deploying
TIMELINE
ProductionCanaryDiffyStaging
@jessicamtai
Localdev

82. Testing&deploying
TIMELINE
ProductionCanaryDiffyStagingLocaldev
Dev
environment
supports shared
dev services
@jessicamtai

83. Testing&deploying
TIMELINE
Replayed
production
traffic with
other staging
services
ProductionCanaryDiffyStagingLocaldev
@jessicamtai

84. Testing&deploying
TIMELINE
ProductionCanaryDiffyStagingLocaldev
Compare
responses from
staging against
production
@jessicamtai

85. RegressionTesting
DIFFY
Staging
(new code)
Primary
(old code)
Secondary
(old code)
Raw response
differences
Non-
deterministic
noise
Filtered
response
differences
Diffy
Replayed traffic
github.com/twitter/diffy
@jessicamtai

86. Testing&deploying
TIMELINE
ProductionCanaryDiffyStagingLocaldev
Deploy to single
instance of
production
@jessicamtai

87. Testing&deploying
TIMELINE
ProductionCanaryDiffyStagingLocaldev
Confidently
deploy to prod
@jessicamtai

88. @jessicamtai

89. @jessicamtai

90. EMPOWERING THE MIGRATION
@jessicamtai

91. 2016:
One small infra team
Airbnb’s SOA progress
@jessicamtai

92. Productculture:
shipthingsquickly
@jessicamtai

93. Orgchallenges:
servicebuildinginparallel
@jessicamtai

94. Product
Frontend
Monorail
Infrastructure
Backend
Monorail+services
Volunteer
sysops
on-call
@jessicamtai

95. @jessicamtai
Product
Frontend
Monorail
Infrastructure
Backend
Monorail+services
Volunteer
sysops
on-call

96. On-callrotationperteam
SERVICE OWNERSHIP
Checkout
service
Pricing service User service
Product Team Infrastructure Team
@jessicamtai

97. PROGRESS SO FAR?
@jessicamtai

98. 2016:
One small infra team
2019:
Whole engineering org
Airbnb’s SOA progress
@jessicamtai

99. ! Faster build & deploy times
○ Hours (Monorail) to minutes (service)
○ Fewer reverts
! Quicker bug fixes
! Increased developer productivity happiness
Promisinginitialresults
SUCCESS
@jessicamtai

100. ! Lower latency from parallelization
○ Ruby monorail single-threaded
○ Java services multi-threaded
! Search results page 3x faster
! Homes description page 10x faster!
Latencyresults
SUCCESS
@jessicamtai

101. Monorailfreeze
@jessicamtai

102. 500+ 67%
Deploys in MonorailEngineers
2016
@jessicamtai

103. 500+ 67%
Deploys in MonorailEngineers
1200+ 7%
2016
2019
@jessicamtai

104. Production traffic via API Gateway
40%
@jessicamtai

105. IDL services in production0
400+
@jessicamtai

106. CheckoutpagerequiredmessageinSOA Checkout presentation
Home data
Reservation data
Review data
Pricing
derived data
Home demand
derived data
Cancellation
derived data
Business travel
derived data
Messaging data
@jessicamtai

107. SOAhasitschallenges
CAUTION
@jessicamtai

108. Distributedservices
CAUTION
@jessicamtai

109. Multiple,isolateddatabases
CAUTION
@jessicamtai

110. Complexserviceorchestration
CAUTION
@jessicamtai

111. ! Prepare for a long commitment
! Decompose incrementally
! Scale with auto-generating frameworks, tools
! Shift development culture
SOAmigration
TAKEAWAYS
@jessicamtai

112. Lookbothways
duringyourGreatMigration
@jessicamtai

113. @JESSICAMTAI
QCON LONDON 2019

114. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
airbnb-services-scalability