The Great Migration: From Monolith to Service-oriented
•
2 likes•36,504 views
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ukrguu. Jessica Tai provides an overview of trade-offs and motivation for the SOA migration. She discusses Airbnb’s architectural tenets around service building and dives deep into lessons learned and best practices developed when undertaking the massive SOA challenge. Filmed at qconsf.com. Jessica Tai has worked at Airbnb for 4 years, starting as a full-stack product engineer for the guest and host booking flow and is now an infrastructure engineer on the Core Services team. She leads the user data service, which is one of Airbnb’s highest QPS services and integrates with all business verticals. She is a member of Leadership & Development committee for women in tech at Airbnb.
Recommended
Recommended
More Related Content
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
The Great Migration: From Monolith to Service-oriented
- 1. Airbnb’s Great Migration: From Monolith to Service-Oriented JESSICA TAI / 11.06.18 / QCON SF
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ airbnb-soa-migration
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 5. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 6. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 7. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 8. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 9. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 10. Life with monolith Growing pains Service design principles Migration to services Best practices Results
- 13. Database model host.first_name, message.save! 2014newhiretask REQUIRED MESSAGE TO HOST
- 14. View template <h1>Tell your host %{host.name}, “Hello”</h1> 2014newhiretask REQUIRED MESSAGE TO HOST
- 15. Controller endpoint logic /submit_booking Model View Controller in Monorail 2014newhiretask REQUIRED MESSAGE TO HOST
- 18. WHY DECIDE TO MIGRATE?
- 19. x Difficulttoscalemonoliths SINGLE MEGA-SERVICE FOR ALL CONCERNS Client ModelA ModelB ModelCModelD ConcernBConcernA ConcernD ConcernC Database Monolith
- 20. Difficulttoscalemonoliths SINGLE MEGA-SERVICE FOR ALL CONCERNS Client ModelA ModelB ModelC ModelD ConcernBConcernA ConcernD ConcernC Database Monolith
- 21. Difficulttoscalemonoliths SINGLE MEGA-SERVICE FOR ALL CONCERNS Client ModelA ModelB ModelC ModelD ConcernB ConcernC Database Monolith ModelE ConcernA ConcernD
- 22. 1,000,000 2,000,000 3,000,000 4,000,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 IncreasinglylargeMonorail LINES OF CODE GROWTH GROWING PAINS
- 23. Growingpains 200+ 200 15h Commits deployed to Monorail per day Engineers at Airbnb Time Monorail deployed blocked per week
- 25. Messagetohost THOUSANDS OF LINES, HUNDREDS OF CONTRIBUTORS Messagemodel
- 28. Oursolution:Service-orientedarchitecture(SOA) NETWORK OF LOOSELY-COUPLED SERVICES Client API gateway Service1 Service2 Database2Database3 Service3
- 29. CheckoutpageinSOA Business travel service Cancellation service Home demand service Pricing service Home service Reservation service Review service Messaging service
- 36. GETTING STARTED WITH THE MIGRATION
- 37. HOSTED BY BAILEY Idyllic home in the trees $99 PER NIGHT
- 38. ! Replace data access methods with service call ! Ruby metaprogramming to override methods Firstattemptsto breakapartMonorail
- 39. Database Home.find_by_host_id(4) ActiveRecord ActiveRecord adapter Mysqladapter Monolith MigratingRails’sActiveRecord select * from homes where host_id = 4
- 40. Monolith CustomActiveRecordadapter Home.find_by_host_id(4) ActiveRecord query object ActiveRecord adapter select * from homes where host_id = 4
- 41. Parsetorequestobject ActiveRecord adapter :type => :select, :table => “homes”, :filters => [{ :name => “host_id”, :type => "integer", :nullable => false, :comparator => :eq, :value => 4, }], :select => [“id”, “host_id”, “title”], Monolith /loadHomes { host_id: 4, fields: [“id”, “host_id”,“title”] } query object
- 42. Re-routequeriestoservices ActiveRecord adapter query object Database Request {host_id: 4 …} Home service Monolith
- 45. Servicetypes STRICT FLOW OF DEPENDENCIES Presentation service Data service Database Derived data service Derived data store Own reads & writes to data entities Shared business logic to disparate data sources Synthesize data from services for end users Middletier Shared validation logic API traffic
- 46. Checkoutpage Checkout page presentation service Reservation data service Reservation database Home demand derived data service Offline booking trend stats Reservationvalidation middle-tierservice Home data service Home database Write Read API traffic
- 48. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B
- 49. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B
- 50. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B
- 51. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 5% Production traffic
- 52. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 10% Production traffic
- 53. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 25% Production traffic
- 54. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 50% Production traffic
- 55. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 100% Production traffic
- 56. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B 100%
- 57. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database Read path A Read path B
- 58. Dualreadcomparison WaitRamp&waitCompareGate Admin UI configuration 1% traffic Gradual increments Gather traffic patterns All traffic through service only Switch Monolith Service Database The read path
- 59. Writecomparison DUAL WRITE TO SEPARATE DATABASES Presentation service Production database Shadow database Write validation middle tier service Write path A Write path B Reads Monolith
- 60. s Writecomparison:services DUAL WRITE Presentation service Production database Write validation middle tier service The write path Monolith
- 62. ! Compare one endpoint at a time ! Unblock clients with incomplete service ○ e.g. /loadUsers ○ fetch users only by id Migratebyendpoint
- 63. Migratebyattribute Service Monolith Database Read migrated attributes Read not-yet-migrated attributes Database Presentation service Production traffic
- 66. Frameworks Auto-generate code Testing&deploying Replay production traffic Observability Standard templates Standardizeservicebuilding CONSISTENCY
- 69. Service Service&clientsetup Business logic Endpoint logic Server transport Java client Ruby client Client transport Client transport
- 70. Service Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Endpoint logic Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking
- 71. Service Endpoint logic Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking
- 72. Service Endpoint logic Service&clientsetup Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking
- 73. IDLusingThrift
- 74. IDL Service Endpoint logic Business logic Server metrics Server diagnostics Startup / teardown Dashboard Dashboard Alert Alert Alert Runbook documentation Metrics Data validation Server transport Server resilience Java client Ruby client Metrics Client transport Data validation Error handling Resilience Metrics Client transport Data validation Error handling Resilience Type checking
- 75. Codingservices BEFORE AFTER ! Difficult to create and maintain services ! Custom Java vs. Ruby clients ! Autogenerated code framework & API ! Automated ruby gem client
- 76. ThriftIDL API FRAMEWORK /* Batch request */ struct LoadSomeDataRequest { 1: optional set<i64> ids (non_null) 2: optional bool fooBar } /* id to data response */ struct LoadSomeDataResponse { 1: optional map<i64, SomeData> data }
- 77. /* /loadSomeData endpoint */ LoadSomeDataResponse loadSomeData (1: LoadSomeDataRequest request) throws (1: SomeException exception1) (accept_replay = "true", rate_limit = "true") ThriftIDL API FRAMEWORK
- 78. Testing&deploying BEFORE AFTER ! Uncertainty in pre-production environments ! Trigger manual requests ! Structured pre-production process ! Automated replayed traffic
- 82. RegressionTesting DIFFY Staging (new code) Primary (old code) Secondary (old code) Raw response differences Non- deterministic noise Filtered response differences Diffy Replayed traffic github.com/twitter/diffy
- 83. Testing&deploying TIMELINE ProductionCanaryDiffyStagingLocaldev Deploy to single instance of production
- 85. Observability BEFORE AFTER ! Nonstandard metrics, dashboards ! Inconsistent alerts ! IDL templated metrics, dashboards ! IDL annotation alerts
- 87. HOW IS THE MIGRATION GOING SO FAR?
- 89. Services using IDL framework 250+
- 90. IDL service endpoints supported 1000+
- 91. ! Faster build & deploy times ○ Hours (Monorail) to minutes (service) ○ Fewer reverts ! Clear service ownership ! Quicker bug fixes Promisinginitialresults SUCCESS
- 92. ! Ruby Monorail single-threaded ! Java services multi-threaded ! Lower latency from parallelization ○ Search results page 3x faster ○ Home description page 10x faster! Latencyresults SUCCESS
- 93. 800+ 3k Deploys per weekEngineers 2017
- 94. 800+ 3k Deploys per weekEngineers 1000+ 10k 2017 2018
- 97. CheckoutpagerequiredmessageinSOA Checkout presentation Pricing data Home data Reservation data Review data Home demand derived data Cancellation derived data Business travel derived data Messaging data
- 102. ! Be ready for a long commitment ! Compare slowly & carefully ! Standardize services ! Frameworks, tools, documentation SOAmigration TAKEAWAYS
- 105. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ airbnb-soa-migration