Russ Johnson of StubHub talks about "Learning Lessons Scaling to 5000 Puppet Agents" at Puppet Camp San Francisco 2013. Find a Puppet Camp near you: puppetlabs.com/community/puppet-camp/
AI You Can Trust - Ensuring Success with Data Integrity Webinar
Lessons I Learned While Scaling to 5000 Puppet Agents
1. Learning Lessons Scaling to
5000 Agents
Russ Johnson
rjohnson@stubhub.com
@professoruss
Confidential Slide 1
2. #whoami
Started out in the mid 90’s
Recovering Windows Admin
Storage Guy
Datacenter Monkey
Once upon a time network guy
At StubHub since December 2006
Working on puppet adoption in a crazy infrastructure
Puppet certified
April 9, 2013 2
7. Build your hosts the same way!
Old way:
systemimager, vmware clones, manual installs
Results:
INCONSISTENCY!
April 9, 2013 7
8. Build your hosts the same way!
New way:
Cobbler < 5m bare metal to on the network
Results:
Same results every time! No drift between base
April 9, 2013 8
9. Set up your working environment properly
Geppetto – eclipse based IDE
http://cloudsmith.github.com/geppetto/index.html
VIM
• Pathogen – For autoloading vim plugins
• Snipmate – Snippets
• Tabular – Text filtering and alignment
• Syntastic – Syntax checking
• mv-vim-puppet – Make vim puppet friendly
• puppet-lint – Syntax checker (gem)
April 9, 2013 9
11. Set up your working environment properly
+
April 9, 2013 11
12. Version Control is not enough
Ever do a 4 way diff across 60 modules to find most of
them different?
April 9, 2013 12
13. What to do?
How do I stop manual edits?
How do I deal with 80+ Dev/QA Environments?
Versioning? Librarian?
Pulp?
Branching?
Internal Forge?
Dynamic Environments?
Puppet Module Tool?
What does PuppetLabs do?
April 9, 2013 13
14. PuppetLabs seems to know what to do
Let’s investigate
puppet module tool
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 14
21. Catching live edits and preventing them
Splunk -> puppet module changes -> alerting
The NOC will hunt you down!
April 9, 2013 21
22. Build and install the module
tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub-
puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/
staging/modules/puppetserver
April 9, 2013 22
24. Internal Forge
mod_rewrite:
Simulate the api – redirect to json metadata files
$htmlroot/api/v1/releases.json?module=user/module
ruby script:
Generate metadata files for each module release and
all modules.
Similar to createrepo (yum)
April 9, 2013 24
29. Avoid case statement insanity
case $::system_role {
‘browse’, ‘search’: {
…do some stuff…
}
‘db’: {
…other stuff…
}
‘otherrole’: {
…please make it stop!!!!
}
}
April 9, 2013 29
30. hieradata
$hieradata/browse.yaml:
---
module::parameter: ‘foo’
$hieradata/search.yaml:
---
module::parameter: ‘bar’
$hieradata/defaults.yaml:
---
module::parameter: ‘I want this everywhere unless there are overrides’
April 9, 2013 30
31. Case -> variables -> hiera
§ 9000 lines of case statements
§ 1000 lines with case/variables
§ ~20 lines with defined type
Code compression FTW!
April 9, 2013 31
33. Release process
§ Syntax check/validate
§ Test on VMs
§ Build module package
§ Release to internal forge
§ puppet module install to staging environment
§ Test again!
§ puppet module install to production environment
April 9, 2013 33
34. The road to yesop
§ Staging
§ Process
§ Repeatability
§ Consistency
§ Document everything
§ Breaking things where it’s cheap
§ Test everything!
April 9, 2013 34
35. Then VS now
§ Environment build time:
– Then: 3+ weeks
• It was wrong
• It didn’t work
• Nobody knew what to expect
– Now: < 1 day
• It’s the same every time
• We know exactly what’s installed
• Internal consumers get what they expect
• Less outages from human error
April 9, 2013 35