This document summarizes Sam Kottler's presentation on using Puppet at large scales. It discusses master-based vs masterless provisioning, certificate authority management, clustering patterns, node classification, external node classifiers, packaging for masterless deployments, distributed runs, deployment practices, and controlled releases. The presentation covers topics relevant to managing massive, multi-datacenter infrastructure with Puppet.

1. Sam Kottler | Puppet at Scale1
Puppet at Scale
Sam Kottler
@samkottler

2. Sam Kottler | Puppet at Scale2
About me
● Worked on large-scale infra for the web @ Venmo,
Acquia, and Digg
● Rubygems.org infrastructure
● Bundler core
● Fedora developer
● Core committer on the Foreman

3. Sam Kottler | Puppet at Scale3
What we'll cover
1. Some basics
2. Master vs. masterless deployment
3. CA management
4. Clustering
5. Node management
6. Development + deployment practices

4. Sam Kottler | Puppet at Scale4
Why we care
● Hyperscale computing
● Massive, multi-DC infrastructure
● Dynamic environments
● The Cloud ™

5. Sam Kottler | Puppet at Scale5
Master vs. masterless

6. Sam Kottler | Puppet at Scale6
Provisioning nodes with a master
1. New node comes online
2. A script is run to install packages and configure
/etc/hosts
3. The agent gets run, generates a CSR, and sends it to
the master
4. The cert gets signed based on an autosign rule or
`puppet cert --sign <nodename>`
5. Puppet runs

7. Sam Kottler | Puppet at Scale7
Provisioning nodes without a master
1. New node comes online knowing its role
2. A script runs to install packages and retrieve
package/tarball
3. puppet apply

8. Sam Kottler | Puppet at Scale8
Certificate authority
● Use by Puppet to authenticate agents
● CSR generated and signed by the CA
● Shared CRL across all CA machines

9. Sam Kottler | Puppet at Scale9
Clustering patterns
● CA has lots of state
● Masters should be stateless
● Reduce the number of file shares

10. Sam Kottler | Puppet at Scale10
DNS-based clustering

11. Sam Kottler | Puppet at Scale11
Load balanced clustering

12. Sam Kottler | Puppet at Scale12
Masters across data-centers
● Shared CA vs. per-region
● Deploy in stages across data-centers

13. Sam Kottler | Puppet at Scale13
Multi-cluster

14. Sam Kottler | Puppet at Scale14
Node classification

15. Sam Kottler | Puppet at Scale15

16. Sam Kottler | Puppet at Scale16
External node classifiers
● Output YAML based on external data
● The Foreman, Puppet Enterprise, Puppet
Dashboard
● Your own custom data source
● Key integration source with your own CMDB

17. Sam Kottler | Puppet at Scale17

18. Sam Kottler | Puppet at Scale18
Packaging for masterless
https://github.com/skottler/librarian-masterless-packaging
● Use /etc/puppet/modules (or modulepath)
● Build RPM's/deb's for distribution
● Publish packages to a repo
● Install/update packages on all machines

19. Sam Kottler | Puppet at Scale19
Distributed runs
● Run puppet based on changes in your code
● Mcollective/SSH/cron

20. Sam Kottler | Puppet at Scale20
Deployment
● Masters are just another deployment target!
● Build CI pipelines
● One-click deployments to masters
● Lint and test your modules

21. Sam Kottler | Puppet at Scale21
cap puppetmaster deploy DC=london

22. Sam Kottler | Puppet at Scale22
Controlled releases
● Separate hosts into groups to do red/black releases
● Build smaller sub-groups of canary hosts
● Monitor your puppet runs

23. Sam Kottler | Puppet at Scale23
Thanks!
@samkottler
shk@redhat.com
https://github.com/skottler