A 15 minute talk I gave at Cloud Connect Chicago 2013 about how and why we use OpenStack at BrightTag

1. OpenStack at BrightTag
Joshua Buss – Cloud Connect 2013
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

2. Intro
Me
● DevOps since before it was called “Dev Ops”
● Orbitz – NOC/SOC
● Graphite specialist
● High-frequency trading
● Capacity planning, DBA, *nix admin, HW geek
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

3. Intro
BrightTag
● World‟s leading Tag Management System
● Globally distributed on AWS
● Server-side tag firing through the cloud
● FUSE profile matching and offline processing
● Next-generation data management
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

4. Motivation
Lay of the land
Four environments:
• Local (each developer‟s laptop)
• Development (in-office only)
• Staging (publicly available, in Rackspace)
• Production (AWS)
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

5. Motivation
The problem
• Production was on an EOL‟d debian
• Staging and dev used same distro
• Each region had separate puppetmaster
• Local is (mostly) OSX (brew; “btiab”)
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

6. Motivation
The fix
• Moved production to ubuntu 11.10
• Centralized puppet repo
• Had to rebuild dev and staging, too
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

7. Motivation
The opportunity
• New dev environment?
• Let‟s make it more like production!
• Time to pick private cloud software
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

8. Process
The goals
• As similar to Amazon as possible
• Open source, active community
• libcloud compatible
• Highly-available
• Web interface
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

9. Process
The choices
• Ganeti
• VMWare
• CloudStack
• Eucalyptus
• OpenStack
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

10. Process
The choices
• Ganeti (no web interface)
• VMWare ($)
• CloudStack (Chaotic at the time)
• Eucalyptus (Tried it, didn‟t work out)
• OpenStack (Tried it, worked great)
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

11. Process
The work
• Found some cheap servers on eBay
• Put Ubuntu 12.04 server on them
• Used our new puppet configs to add users, etc.
• Followed the docs
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

12. Process
The work
• Found some cheap servers on eBay
• Put Ubuntu 12.04 server on them
• Used our new puppet configs to add users, etc.
• Followed the docs
• Yeah, right.
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

13. Process
The work
• Took about a month to learn:
• Terminology
• How all the pieces fit together
• Highly available networking support
• Routing and ARP tables
• Thank you, IRC
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

14. Result
The finished product
• Hey look, you can use this nice UI
• ...or libcloud
• ...and our single puppet master
• ...and the result is just like our production instances!
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

15. Result
Open Stack was just the start
• Dev environment now has over 50 VMs on 15 servers
• Standardize all the environments!
• ...even some new CI environments
• ...and our local environments (virtualbox)
• Jenkins is integral in our continuous deployments
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

16. Thank you!
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

17. Extras
BT Infrastructure
• Many small, HTTP webapps working together
• Key off „server groups‟ to define each environment
• “data01we2” - Server group “data” in the “we2” region
"roles": {"we2": {"data": ["dataserve", "auxserve"] } …
• Roles define which puppet modules we install
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

18. Extras
Zerg
• libcloud + flask
• Provisioning, real-time CMDB, generated configs
• HAProxy, Nagios, Cassandra configs
• “Manifest” - JSON that include IP Addresses
• Use manifest to know where to deploy each app
• Use zerg for DNS as well (updating /etc/hosts)
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

19. Extras
Puppet
• Source-controlled through multiple branches
• Puppet master has each branch checked out into a different directory
• Branches are pulled automatically
• Each branch has a matching puppet „environment‟ name
• /etc/puppet/puppet.conf “environment” determines the client
• All environments, including our guest VMs, default to master
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

20. Extras
Zero-downtime deploys
• /bthc of each webapp determines healthiness of app
• Fabric puts each app into maintenance mode before stopping
• App stops taking traffic, then gets restarted with new code
• New app doesn‟t respond with „healthy‟ until its actually ready for traffic
• When there are a lot of an app, we parallelize the deploy
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG