Gilt uses Puppet for configuration management across its infrastructure. It has around 1000 Puppet modules and uses tools like Puppetmaster, Puppet Dashboard, and an in-house External Node Classifier called Mothership. Changes are rolled out incrementally using a development, canary, and production process. Mothership also handles provisioning, assets management, and mapping hosts to Puppet modules. Lessons learned include keeping modules small and simple, planning for change like OS upgrades, and using multiple views in Mothership for logical vs physical infrastructure. Future plans aim to accelerate incremental deployments daily with easy rollbacks while maintaining 100% uptime.

1. Puppet Camp
Dublin - 2012
Andrés Pereira <apereira@gilt.com>
Roland Tritsch <rtritsch@gilt.com>

2. Gilt in a nutshell ...
● 5, 500, 1000/200, 300, 1, 18, ...

3. Gilt in a nutshell ...
Frontend (HTML[5], CSS, JS, ...) ● 2 DCs + AWS EC2
● CentOS Linux
Load Balancing (Zeus) ● CDN (Akamai)
● Puppet, Cobbler, ...
Aggregation (Java, Scala, ...)
● Zenoss, Nagios, ...
● Git & Gerrit
Load Balancing (Zeus)
● Jenkins
Services (Java, Scala, Ruby, ...)
Database (PostgreSQL, MongoDB, ...)

4. Puppet at Gilt ...
● We've been using Puppet for almost 3 years
○ ~ 1000 puppet modules
● Typical setup
○ Apache httpd+passenger for Puppetmaster
○ Puppetcommander (MCollective plugin) to schedule
puppet runs
○ Puppet Dashboard (Reporting)
○ In-house External Node Classifier (aka 'Mothership')

5. Puppet - Incremental rollout

6. Puppet - Incremental rollout
● Deployment of changes
○ Commit to 'development' branch
○ Push changes for code review
○ Once changes have been reviewed and verified,
they get merged into 'development'
○ A 'canary' flag can be set for a number of nodes
○ Change is cherry-picked to 'master' branch
○ A 'tag' from master is created and deployed
■ See behavior in the canary environment
■ Expand the canary env. Rinse & repeat
■ Remove canary flag

7. Puppet - External node Classifier
http://mothership.sourceforge.net/

8. Puppet - External node Classifier
● Mothership
○ In-house assets management & provisioning tool. It
also acts as an External Node Classifier for Puppet
■ Can provision bare-metal & virtual machines
■ Integrated with Cobbler
■ Also manages users/groups & sudoers
● Synchronized to LDAP
● Puppet deploys sudoers file
■ DNS management
■ Hosts can have one or more labels (tags) that
are mapped to Puppet modules

11. Lessons learned / Best practices ...
Puppet ...
● Don't just run it!
● Node mgmt can become difficult (+100)
● Keep change in mind (e.g. OS upgrades)
Mothership ...
● ... is not enough. You need at least 2 views
○ Logical (deployed/reserved), Physical
● You need to live with Physical != Logical
○ Physical will be generated. It is a feedback loop.

12. Next steps ...

13. Requirements ...
● Accelerate our ability to do incremental
deployments (multiple times a day)
○ ... with easy rollback
○ ... accelerating our ability to innovate
○ ... while maintaining 100% uptime
● The "Happy Path" needs to be without
human intervention
○ From Commit to A/B test

14. Architecture ...
QA - Selenium / Deployment -
Code - Gerrit
TestComplete Puppet
Ion Cannon
Build -
Performance Analytics
Jenkins/Nexus

15. Takeaways ...
● Keep your modules small, keep them simple
● Keep change in mind
○ Do incremental rollout of changes
○ Provide a 'default' case in your puppet modules and
use 'fail' (sdtlib) as a default case.
● If you are looking for a complete CD
platform, then you need to embed Puppet
into a larger solution

16. Questions?