This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation
Sam Bashton
Director, Bashton Ltd
After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.
4. ABOUTME
Linux guy since Slackware, floppy disks and root + boot
Using Puppet since 2007
Run a company in Manchester, North West England
We provide outsourced ops for other companies
6. HOWWEWORK
Simple is better than complex
Complexity is worth adding only if it provides obvious
functional benefits
Re-usability
Resilience
7. WHY DID WE PICK AWS?
Featureset and toolset massively in advance of any
other cloud provider, public or private
#1 customer reason for switching to AWS? The ability to
scale on demand
8.
9. TOOLSWEUSEFOR
BUILDINGANDMANAGING
Do one thing and do it well
CloudFormation - Amazon tool to manage
infrastructure
Puppet - Manage system configuration
Pulp - centralised repository, manages package
revisions
Jenkins
15. NO MECHANISMFOR
CLEANINGOLDHOSTS
Likely to have host-names reused, causing machines to
fail to configure
Puppetmaster will fill with certificates for machines that
ran for a few hours and went away again
16. POTENTIAL WORKAROUNDS:
Use UUID certificates
Agree not to look in the certificate directory
Write mechanism for cleaning up old certificates
18. POTENTIAL WORKAROUNDS
Use an external node classifier
Use some mechanism for giving a better hostname, eg
web-172-26-5-123 and use regex for node names
26. APPLYPUPPETMANIFESTS
Have an RPM %postinstcommand apply the Puppet
config
This isn't as straightforward as running the puppet
applyfrom %postinst
Puppet needs to install packages via yum, but yum is
running installing the Puppet package
Instead, we work around with a dirty hack: have the
%postinstcreate an atscript which checks if yum
has finished and then runs the puppet apply
28. PULP
We were already using Pulp
Provides yum repository management
Used for managing security updates and deploying
application code
http://pulpproject.org/
29. WHATISPULP
Repository manager
Allows us to easily audit what packages and versions
are installed where
Allows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easy
replication and clustering
30. HOWWEUSEPULP
Puppet contains details of what packages should be
installed
Pulp manages which version of the package should be
installed
Pulp allows us to clone repos and copy packages
between them for easy qa->stage->live environment
management
33. DETAILSON HOWWE
DEPLOYCODE
Jenkins fetches code from source control (git)
An RPM is built
Tests are run
If tests pass, the RPM is added to the relevant Pulp
repository RPM installed on the target machine(s)
34. DEPLOYMENTLIFE-CYCLE
Jenkins also manages deployment life-cycle
RPMs are installed on staging
Promoted Builds plugin then used to install the same
RPMs on live once testing is complete
35. PUPPETDEPLOYMENT
PROCESS
Puppet manifests are checked into git
Lint tests via
Jenkins pulls in modules with librarian-puppet, then
builds an RPM
Deployment to test environments, functional tests for
wider code-base run
Jenkins Warnings plugin
36. PUTTINGITINTO
PRODUCTION
Once suitable tests (automated and manual) have been
carried out, we promote Puppet config into production
We use the Jenkins 'Promoted Builds' plugin for this
38. EXCEPT..
How does a machine get from a bare image to the state
where we can push packages to it from Pulp?
How does a machine know what type of machine it is?
How do we find other resources, eg database
hostname?
39. CLOUDFORMATION
Amazon tool for specifying infrastructure
Everything* we provision inside AWS is provisioned via
CloudFormation
JSON templates
* Everything except for the things Amazon doesn't expose
via CloudFormation..
40. CLOUD-INIT
Works with multiple cloud types
Sorts out things like SSH keys, allows us to configure
host names
Also allows us to provide a bash script to run on startup
41. PROVISIONINGABARE
INSTANCE
cloud-init automatically manually adds the pulp repo
which contains Pulp, Puppet and our Puppet
manifests/modules
Installs appropriate RPMs
Puppet runs, subscribing the machine to the relevant
Pulp repos, and installing packages in the usual Puppet
way
43. EXTRAFACTS
Custom facter facts
Also specified in an environmental variable
Data comes from within the CloudFormation template
On our list of things to look at:
FACTER_HOSTENVIRONMENT=live
FACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hiera-cloudformation
44. OTHER RESOURCES
We either:
Provide details as a facter fact
`FACTER_DBHOST=xyz
Also use this approach to limit distribution of
secure details, eg DB passwords
Discover via the EC2 API
Eg Varnish servers discover web backends by
calling API and finding hosts tagged appropriately
46. FREEWINS!
Greater control over the timing of Puppet runs
Improved visibility - for ops and devs
Configuration changes now have to be deployed to
testing/staging first
47. MOREFREEWINS!
Puppet configs now have a version
Easy to find config version on the machine itself
Config changelogs accessible on every machine
(Git changelog added to RPM)
48. THEDOWNSIDES
Puppet manifests and modules on all machines
Potentially a security issue?
Mitigated by CloudFormation holding most sensitive
data
49. ALTERNATIVE
IMPLEMENTATIONS
Don't want to use Pulp?
Could do basically the same thing with yum s3 plugin
Use mcollective to push package updates
https://github.com/jbraeuer/yum-s3-plugin
50. FUTUREIMPROVEMENTS
Build AMIs using Packer instead of configuring at boot
time
Decrease time to autoscale
Would probably still need to run Puppet at first boot
to configure machine specific settings