The presentation that I gave at the 'NYC Tech Talks' meetup @ January 14, 2014

1. SCALING GILT
From Monolith Ruby App
to
Distributed Scala Micro-Services
#NYCTECHTALKS
Yoni (Jonathan) Goldberg
Lead Engineer - GILT
Podium => http://bit.ly/podiumapp

2. ABOUT ME
- Leading the Popeye Team
- Sale Personalization, Loyalty, SEO
Post-purchase, Login/Registration flows
- MIT CS BS/Meng | Google | IBM | IDF
- Brooklyn | Coffee | Arduino | Running | Kite Surfing |
Online Collaboration | Poker
Excited to be part of the NYC Tech community

3. THE LESSONS AND CHALLENGES THAT
WE HAD/HAVE WITH
MICRO-SERVICE ARCHITECTURE

5. WHAT IS GILT?
Flash Sales Business
Founded in 2007
Top 50 Internet-Retailer
~150 Engineers

7. ANOTHER WAY TO LOOK AT GILT
Three day traffic pattern

8. THE CLASSIC
STARTUP STORY

9. THE EARLY DAYS
2007 - Ruby on Rails the hottest new thing
The goal was to get to market fast

11. WE WERE ABLE TO HANDLE OUR
TRAFFIC PRETTY WELL

12. UNTIL LOUBOUTIN CAME TO GILT

13. TECHNOLOGY PAIN POINTS - 2009
Spike required to launch 1,000s of ruby processes
Postgres was overloaded
Routing traffic between ruby processes sucked
|Note to self| - hide from the ruby fan boys

14. DEV PAIN POINTS
1000 Models/Controllers, 200K LOC, 100s of jobs
Lots of contributors + no ownership
Difficult deployments with long integration cycles
Hard to identify root causes

15. WE NEEDED TO SOLVE
THE PROBLEM FAST

16. THREE THINGS HAPPENED
Started the transition to the JVM
M(a/i)cro-Service Era Started
Dedicated data stores

17. WHY JVM?
Widely adopted
Stable
Better support for concurrency
Better GC vs MRI

18. FIRST 10 SERVICES

20. We solved 90% of our arch scaling problem
But not the Dev points

21. PAIN POINTS
Spike required to launch 1,000s of ruby processes
Postgres was overloaded
Routing traffic between ruby processes sucked
New services became semi-monolithic
1000 Models/Controllers, 200K LOC, 100s of jobs
Lots of contributors + no ownership
Difficult deployments with long integration cycles

22. WHY WE DOUBLED DOWN ON
MICRO-SERVICES
Empower teams and ownership
Smaller scope
Simpler and Easier deployments and rollbacks

23. MICRO SERVICE ARCHITECTURE
STARTED TO GET TRACTION

24. AS OF LAST WEEK WE HAVE MORE
THAN
450 SERVICES

25. APP BOOTSTRAP
rk bosrpamnwb
ae otta:di-e
#Bosrpaamnwbsrie
otta
di-e evc
rk bosrpbblndc
ae otta:ayo-os
#Bosrpabblndc srie
otta
ayo-os evc
rk bosrpcin-evrcr #Bosrpacin-evrcr sr
ae otta:letsre-oe
otta
letsre-oe ev
rk bosrpjre-aa
ae otta:esyjv
#Bosrpajre-aasrie
otta
esyjv evc
rk bosrpjre-cl
ae otta:esysaa
#Bosrpajre-cl srie
otta
esysaa evc
rk bosrppa
ae otta:ly
#Bosrpapa srie
otta
ly evc
rk bosrppa-ibid
ae otta:lyu-ul
#Bosrpapa-ibidsrie
otta
lyu-ul evc
rk bosrpstlbay
ae otta:b-irr
#Bosrpastlbaysrie
otta
b-irr evc
rk bosrpshm
ae otta:cea
#Bosrpashm srie
otta
cea evc

26. WE BEGAN THE TRANSITION TO SCALA
AND PLAY
LOSA - Lots Of Small Apps
Same motivation and benefits of Micro-Service
Architecture

28. NEW CHALLENGES
Dev/Integration Environments
Who owns this service!?
Monitoring
Deployments and Testing (Functional/Integration)

29. ON DEV/INTEGRATION ENVIRONMENTS
The hardware is not strong enough
No one wants to compile 20 services

30. EACH TEAM HAS A STAGING ENV
SRIEPRS[
EVC_OT=
40,#itn-evc
01 lsigsrie
83,#v-srst
25 scue-e
92,#v-refl
40 scfe-al
79,#v-oat
85 scLyly
85,#e-oat
15 wblyly
91,#e ivnoysau
40 wb netr tts
79,#di-oat
88 amnlyly
79,#oiiain
89 ntfcto
70,#og
12 rue
93,#v-opnn
50 sccmoet
60,#v-atitsbi
82 scwils-umt
46,#v-cinsl
06 scato-ae
..
..
PR_OWR_RSSRIEPRSmp{|ot
OTFRADAG=EVC_OT.a
pr|
[-' "{ot:oahs:{ot"
'L, #pr}lclot#pr}]
}
ee([wsh- - - -} PR_OWR_RS G_OT.l
xc*%{s a C N n, OTFRADAG, WHS]f

31. STAGING DIFFICULTIES:
Hard to keep all the services up to date
Maxed our staging env capacities
Requires to have internet connection for some of the
services (e.g LOSA-apps)

32. The Future

33. DOCKER
An extension to Linux Containers (LXC)
Decentralization
Simple Configurations
Much lighter than a VM
Immutable
Supports services and platforms

34. ON OWNERSHIP
"code stays much longer than people" - SB

35. CODE OWNERSHIP

36. CURRENT APPROACH
Code Review!Code Review!Code Review!
Team owns services, not individual developers
Ownership transfer

39. DATA OWNERSHIP

40. WE TRANSITIONED TO MICRO-DBS
Third of the services have their own
MongoDB
Postgres
Voldemort

41. MANAGE MICRO-RELATIONAL DBS
SCHEMA EVOLUTION MANAGER
https://github.com/gilt/schema-evolution-manager

42. PRINCIPLES OF
SCHEMA EVOLUTION MANAGER
Can manage the schema evolutions in a Git repo
Schema changes are deployed as tar flies
No rollbacks
Schema changes are required to be incremental

43. eh "raetberlae (ditgr">nwsl
co cet al eess i nee)
e.q
smad.nwsl#rae agtcmi
e-d /e.q Cetd
i omt
smds #gnrtstetregshm-o-ann...a.
e-it
eeae h a . ceaincno002tr
#Spadutro yu sre
c n na n or evr
c shm-o-ann002
d ceaincno-..
smapyhs -lclot-nm incno -ue incn
e-pl ot -oahs -ae o-ann -sr o-an

44. ON MONITORING

46. THE TOOLS WE USE
graphite / openTSDB

52. ON DEPLOYMENTS
AND TESTING
(FUNCTIONAL/INTEGRATION)
"Testing is HARD" - the dev that sits on your left

53. THE CHALLENGES THAT WE FACED:
Hard to execute functional tests between services
Frustrating to deploy semi-manually (Capistrano)
Scary to deploy other teams services

54. SBT
Motivation: Scala adaption
Complex Scala syntax
Cool features: ~test, shell, console
Hard to debug

55. GILT-SBT-BUILD
Simple config for all the services
Pulls many plugins:
[nexus, testing, RPMs, run scripts, Monitoring,
SemVer, ...]
Custom commands (e.g 'sbt release')

56. ojc BidetnsCinSreCrPoetwt Dpnece {
bet ul xed letevroerjc ih eednis
vlnm ='v-aeatvto'
a ae
scsl-ciain
vlcrDp =..
a oees
..
vlsreDp =..
a evres
.
vlcinDp =..
a letes
.
oerd vlinannrc =InannFsTak
vrie a ocnoTak
oCno.atrc
}

57. ION-CANNON + SBT
Run functional/Selenium tests on dedicated Env
Supports Canary releases
Easy rollbacks
Integrated health checks

59. MAIN TAKEAWAYS
Simplicity - Do you really need it?
We feel that it was the right choice for us
MicroServices promise works for most cases
As of 2014 - You will need to invest in Tools!

60. PODIUM TIME
We are hiring...
Keep in touch:
jgoldberg@gilt.com
www.yonigoldberg.com