This document summarizes Yoni Goldberg's talk on Gilt's transition from a monolithic Ruby on Rails application to a distributed system of over 450 Scala microservices. Some of the key challenges they faced included scaling issues as traffic grew, difficulties deploying and testing changes across many interdependent services, and a lack of clear ownership over parts of the codebase. Gilt addressed these by moving to the JVM, embracing microservice architecture, implementing tools like Docker and a custom SBT build to simplify deployments, and establishing clear ownership for each microservice.
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
Scaling from Monolith to Microservices
1. SCALING GILT
From Monolith Ruby App
to
Distributed Scala Micro-Services
#NYCTECHTALKS
Yoni (Jonathan) Goldberg
Lead Engineer - GILT
Podium => http://bit.ly/podiumapp
2. ABOUT ME
- Leading the Popeye Team
- Sale Personalization, Loyalty, SEO
Post-purchase, Login/Registration flows
- MIT CS BS/Meng | Google | IBM | IDF
- Brooklyn | Coffee | Arduino | Running | Kite Surfing |
Online Collaboration | Poker
Excited to be part of the NYC Tech community
3. THE LESSONS AND CHALLENGES THAT
WE HAD/HAVE WITH
MICRO-SERVICE ARCHITECTURE
4.
5. WHAT IS GILT?
Flash Sales Business
Founded in 2007
Top 50 Internet-Retailer
~150 Engineers
13. TECHNOLOGY PAIN POINTS - 2009
Spike required to launch 1,000s of ruby processes
Postgres was overloaded
Routing traffic between ruby processes sucked
|Note to self| - hide from the ruby fan boys
14. DEV PAIN POINTS
1000 Models/Controllers, 200K LOC, 100s of jobs
Lots of contributors + no ownership
Difficult deployments with long integration cycles
Hard to identify root causes
20. We solved 90% of our arch scaling problem
But not the Dev points
21. PAIN POINTS
Spike required to launch 1,000s of ruby processes
Postgres was overloaded
Routing traffic between ruby processes sucked
New services became semi-monolithic
1000 Models/Controllers, 200K LOC, 100s of jobs
Lots of contributors + no ownership
Difficult deployments with long integration cycles
22. WHY WE DOUBLED DOWN ON
MICRO-SERVICES
Empower teams and ownership
Smaller scope
Simpler and Easier deployments and rollbacks
30. EACH TEAM HAS A STAGING ENV
SRIEPRS[
EVC_OT=
40,#itn-evc
01 lsigsrie
83,#v-srst
25 scue-e
92,#v-refl
40 scfe-al
79,#v-oat
85 scLyly
85,#e-oat
15 wblyly
91,#e ivnoysau
40 wb netr tts
79,#di-oat
88 amnlyly
79,#oiiain
89 ntfcto
70,#og
12 rue
93,#v-opnn
50 sccmoet
60,#v-atitsbi
82 scwils-umt
46,#v-cinsl
06 scato-ae
..
..
PR_OWR_RSSRIEPRSmp{|ot
OTFRADAG=EVC_OT.a
pr|
[-' "{ot:oahs:{ot"
'L, #pr}lclot#pr}]
}
ee([wsh- - - -} PR_OWR_RS G_OT.l
xc*%{s a C N n, OTFRADAG, WHS]f
31. STAGING DIFFICULTIES:
Hard to keep all the services up to date
Maxed our staging env capacities
Requires to have internet connection for some of the
services (e.g LOSA-apps)
33. DOCKER
An extension to Linux Containers (LXC)
Decentralization
Simple Configurations
Much lighter than a VM
Immutable
Supports services and platforms
42. PRINCIPLES OF
SCHEMA EVOLUTION MANAGER
Can manage the schema evolutions in a Git repo
Schema changes are deployed as tar flies
No rollbacks
Schema changes are required to be incremental
43. eh "raetberlae (ditgr">nwsl
co cet al eess i nee)
e.q
smad.nwsl#rae agtcmi
e-d /e.q Cetd
i omt
smds #gnrtstetregshm-o-ann...a.
e-it
eeae h a . ceaincno002tr
#Spadutro yu sre
c n na n or evr
c shm-o-ann002
d ceaincno-..
smapyhs -lclot-nm incno -ue incn
e-pl ot -oahs -ae o-ann -sr o-an
53. THE CHALLENGES THAT WE FACED:
Hard to execute functional tests between services
Frustrating to deploy semi-manually (Capistrano)
Scary to deploy other teams services
55. GILT-SBT-BUILD
Simple config for all the services
Pulls many plugins:
[nexus, testing, RPMs, run scripts, Monitoring,
SemVer, ...]
Custom commands (e.g 'sbt release')
56. ojc BidetnsCinSreCrPoetwt Dpnece {
bet ul xed letevroerjc ih eednis
vlnm ='v-aeatvto'
a ae
scsl-ciain
vlcrDp =..
a oees
..
vlsreDp =..
a evres
.
vlcinDp =..
a letes
.
oerd vlinannrc =InannFsTak
vrie a ocnoTak
oCno.atrc
}
57. ION-CANNON + SBT
Run functional/Selenium tests on dedicated Env
Supports Canary releases
Easy rollbacks
Integrated health checks
58.
59. MAIN TAKEAWAYS
Simplicity - Do you really need it?
We feel that it was the right choice for us
MicroServices promise works for most cases
As of 2014 - You will need to invest in Tools!
60. PODIUM TIME
We are hiring...
Keep in touch:
jgoldberg@gilt.com
www.yonigoldberg.com