A case study of the tools and techniques used at Gilt Groupe to develop and deploy a system composed of over 200 micro-services.

1. INNOVATING FASTER
Supporting microservices w/ SBT, Continuous Delivery & LXC
Kevin Scaldeferri
Gilt Groupe
June 18, 2013

4. CULTURE
• Rapid feature development
• Rapid experimentation
• Find what works, discard what doesn’t

5. INTHE BEGINNING

6. ONE (REALLY) BIG RAILS APP
• ~1000 models and controllers
• ~ 200k lines of Ruby
• ~ 50k lines of PostgreSQL stored procedures

7. PROS OF MONOLITHIC
DESIGN
• Rapid (Initial) Development
• Simple development
• Simple deployment
• Simple operational model

8. CONS OF MONOLITHIC
DESIGN
• Unclear Ownership
• Complex Dependencies
• LengthyTest Cycles
• Unexpected Performance Impacts

9. DAY OF RECKONING

10. THE LOUBOUTIN INCIDENT

11. GETTING
(THE MICROSERVICES)
RELIGION

12. EARLY DAYS
• Pure Java
• Moving fast, lots of C&P
• Lots of snowflakes and competing approaches
• Ant-based build installed via git submodule
• Capistrano for deployment

13. DIGRESSION ON GIT
SUBMODULES

14. DOWNSIDES OF ANT
• Build scripts in a different language
• Hard to integrate many tools
• Impedance mismatch w/ Scala

15. SBT

16. WHAT’S COOL ABOUT SBT
• Fast incremental builds of Scala & Java
• ~compile, ~test
• Interactive shell & console
• Configuration is “just” Scala code

17. BUT...
• Very sophisticated use of Scala
• Subtle mental model
• Can be hard to debug
• Lots of historical baggage in our build & deployment processes

18. GILT-SBT-BUILD
• One big SBT plugin pulling in all other plugins and
configuration
• Plugin is versioned just like any other software
• (Actually builds itself!)
• Super simple config for individual services

19. gilt.GiltProject.jarSettings
name
:=
"lib-­‐jenkins"
libraryDependencies
++=
Seq(
"net.databinder"
%%
"dispatch-­‐core"
%
"0.8.8",
"net.databinder"
%%
"dispatch-­‐http"
%
"0.8.8"
)

20. PLUGIN PROVIDES
• Nexus config
• Testing & Coverage Libraries
• RPMs
• Standard Run Scripts
• NewRelic Support
• Release Emails
• SemVer Analysis
• Add’l Dependency
Heuristics
• Integration Builds
• ... and more....

21. WHAT ABOUT MULTI-
PROJECT BUILDS?

22. object
Build
extends
ClientServerCoreProject
{
val
name
=
"svc-­‐persistent-­‐session"
val
coreDeps
=
...
val
serverDeps
=
...
val
clientDeps
=
...
override
val
ioncannonTrack
=
IonCannon.FastTrack
}

23. IONCANNON?

24. CONTINUOUS DELIVERY
• At the granularity of released versions
• Too many commits and too many services to deploy every
commit

25. DEVELOPMENT
• Teams have a shared dev / test environment
• Develop locally w/ tunnels to services
• Submit to Gerrit for peer review
• Deploy & test on team environment
• ‘sbt release’ when ready

26. FAST-TRACK
• Fully automated testing & deployment
• ‘sbt release’ triggers deployment to a staging environment
mirroring production
• Automated tests run
• Promote to production or rollback

28. SKYPE NOTIFICATIONS

29. FAST-TRACK
Launched in November 2012. Currently over
100 services and applications.

30. SLOW-TRACK
• Automated Deployment
• ManualTesting / Approval

32. SLOW-TRACK IN PRACTICE
• Dramatically less adoption than fast-track
• Systems without good automated tests tend to be older
systems which also haven’t upgraded to the current build &
deploy toolchain
• Teams do manual testing on test environments already
• Better tools for UI testing tend to eliminate the need for this

33. SELENIUM:TEST
• ‘sbt selenium:test’
• Built on ScalaTest 2.0 Selenium DSL
• Automated browser testing with reusable components
• All tests written in Scala
• Tests can be run in any environment
• Selenium grid available

34. @Selenium
class
Example
extends
FlatSpecTestBase
with
Matchers
with
ConfigurableBrowser
with
LoggedTestUser
with
OnProductDetailPage
with
AvailableToPurchaseItem
with
InMens
with
CloseBrowserAfterAllTests
{
"A
size
chart"
should
"be
available"
in
{
//
test
goes
here
}
}

35. DEPLOYMENT

36. AN APOLOGY

37. CURRENT DEPLOYMENT
• Bare metal machines
• Multiple services per machine
• Highly manual process to balance resource requirements

38. THE FUTURE
• Deploy to immutable LXC containers, one service per
container
• New versions deploy to fresh containers, old containers are
retired
• Services specify resource requirements, system finds available
capacity

39. CLOUDSTACK
• Private cloud management
• Also evaluated OpenStack & Ganeti
• CloudStack’s APIs seemed best suited to our design
• Added LXC support to CloudStack 4.2

40. WHY LXC?

41. DEPLOYMENT SEQUENCE
• svc-software-provisioning - create new container(s)
• install new version onto containers
• svc-loadbalancer - move traffic to new version
• svc-software-provisioning - delete old containers

42. SUMMARY
• Support rapid development of micro-services and micro-
applications via:
• Powerful & highly abstracted build system
• Continuous delivery & great tools for automated testing
• (Upcoming) Simple & consistent hardware provisioning

43. THANKYOU

44. QUESTIONS?