Puppet Camp Chicago 2014: "Puppet at backstop another year of lessons" by Bill Weiss, Backstop

1. Puppet at Backstop
Another year of lessons
1

2. $ whoami
‣ Bill Weiss <bill@backstopsolutions.com>
‣ @BillWeiss
‣ Probably responsible for 90% of the badness in
this talk, 25% of the good
‣ Still not a designer
2

3. What’s this talk about?
aka, who cares?
3

4. 4
Image via MemeCrunch. Original owned by the History Channel.

5. No, wait, that’s not it
(Come talk to me afterward about that)
5

6. ‣ I’m here to talk about things that work (and don’t)
for us in our environment
‣ Hopefully most of it is useful to you
‣ Some might just be there so you can point and
laugh
6

7. –You
“What the heck is Backstop?”
7

8. Super helpful, I know
8

9. SaaS for institutional
investment firms
‣ That’s it.
‣ You’re probably not our target audience
‣ If you are, awesome, let’s talk later
9

10. Some numbers
‣ ~195 Puppeted machines in production (US)
‣ ~350 total
‣ 2 datacenters, 2 AWS AZs, 1 office closet full of
machines
‣ 50+ Engineering folks (IT, dev, QA, etc)
Numbers of machines from MCollective!
10

11. Logo owned by the CentOS project
Logo also owned by Oracle, of course
Logo owned by Oracle
Logo by David Heinemeier Hansson
11

12. ‣ Big application is J2EE on JBoss
‣ Lots of smaller bits (“services”) on Tomcat
‣ Second app is Ruby on Rails
12

13. How ‘bout that Puppet?
‣ Moved to GitHub 2013/11/18
‣ ~4100 commits since then (~15 commits/day!)
‣ 2700 from people outside sysadmin team
‣ ~1350 pull requests from 21 contributors
13

14. The move to GitHub was
awesome
‣ 4 years of Subversion before that had 29 authors
and 6700 commits
‣ Other things at play here beside just s/svn/git/ ,
but it helped a lot
‣ How?
14

15. ! git workflow
‣ Org-owned repo, two branches
‣ master ➡ production
‣ office ➡ testing (pre-prod)
‣ Everyone forks from master and sends pull
requests to office
‣ Pull changes from office to master as needed
15

16. ‣ Everyone has their own fork with work branches
‣ Branch names probably are Jira ticket numbers
‣ Everyone bases from master, not office, so we
can pull changes independently
16

17. This is why we have
so many contributors
17

18. ‣ If you’re a software shop, you already have a
workflow. Do that
‣ Otherwise, this is working well for us
18

19. " git hooks
‣ We had a pretty good set of pre-commit hooks in
svn to test changes
‣ Github doesn’t do that #
‣ Git pre-commit hooks exist, but it’s up to each
person to set them up
‣ No awesome answer yet
‣ The answer might be…
19

20. ! Jenkins!
‣ We used to have this “deploy” job on the Puppet
masters as a cron job:
• (cd /etc/puppet && svn up)
‣ Nothing could go wrong there, right?
‣ For a while we just did it manually instead
20

21. ‣ Jenkins polls for changes to either branch
‣ Runs a deploy job that grabs repo, runs
librarian-puppet, a “build” step, and then pushes
it out to the masters
‣ Determines masters via mco call
‣ All the good (and bad) of capistrano for the push
21

22. 22
alias puppet-test='sudo puppet agent --test --noop --
environment'
That deploys an environment titled UserBranch, camel-cased

23. ‣ Jenkins also runs a test suite in parallel
‣ Some day that’ll be a precursor to the deploy
job. Some day…
23

24. " Sometimes you just want to
do something to prod
‣ All that well-defined procedure means time for
quick fixes goes up
‣ Make a branch, make some changes, commit,
push, PR to office, merge, PR to prod, merge,
run deploy…
24

25. ‣ Possible answers here:
• Pull some things out into their own repos
• Accept that it’s slower
• Tool it up, make the machines do the slow
parts
25

26. ! MCollective is still awesome
26
Original image: http://xkcd.com/353/
Cheapo editing courtesy OS X Preview “Annotate”
Same graphic as last year’s talk. I love you, mco!

27. It’s like Frank’s Red Hot
‣ Our deploys (not IT, app teams) now use it to
figure out which machines to deploy to
‣ We wrote an agent for our VM system (Proxmox)
Which, uh, evidently we haven’t released. Will do.
‣ Stop/start the world for Oracle maintenance
27

28. " ActiveMQ is hard
‣ MCollective sits on top of ActiveMQ
‣ Easy, right? Just install it (with Puppet, even)
and go to town
‣ Second datacenter? Yeah, just get them talking
to each other and…
28

29. A sad example
29

30. Suddenly, one day…
‣ Pretend x is time
‣ y is memory usage
‣ That’s our activeMQ
• At 2AM
• Every day
30

31. 31
“Sudden Clarity Clarence” via memecreator.eu
Original image? Who knows. Might be Clarence.

32. ‣ Long story short: make really sure your broker
names are unique across your network
‣ Really really sure
32

33. That won’t happen to us
‣ Maybe not
‣ Just know you’re adding another service you rely
on
‣ Monitor it
‣ Measure it
‣ Have a game plan for what to do when it goes
horribly wrong
33

34. ‣ R.I.’s scripts [1]
‣ Erwan Ben Souiden, aka labynocle [2]
‣ A script I butchered to help
34
[1]: https://github.com/ripienaar/monitoring-scripts/tree/master/activemq
[2]: https://github.com/labynocle/nagios_plugins_by_labynocle/tree/master/check_activeMQ
[3]: https://github.com/BillWeiss/MiscScripts/blob/master/check_activemq.pl
I know, I know, bleh, Nagios

35. ! Cloudstack!
‣ It’s great
‣ Setup was easy
‣ Only using it in test now (so we only kinda care if
the machines burn down)
35
That’s the CloudStack monkey, owned by the Apache Software Foundation

36. ‣ Remember that puppet-test alias?
‣ That + CloudStack means people can build
machines to test Puppet changes to
‣ I was surprised recently by a service that just
showed up ready to run in prod, Puppet and all
36

37. You don’t have to use CloudStack,
but you need something like it
‣ That could be OpenStack, Vagrant, even some
AWS credit
‣ You just need fast to build disposable machines
37

38. " Would be better in prod
‣ We’re not yet running CloudStack in production.
It’ll happen someday.
‣ So, while I’m singing its praises, I can’t say we’re
all in.
38

39. ! create_resources
‣ I have to show off one chunk of code
‣ Let me quote from TFM:
39
‣ That’s vague, right? The example in TFM didn’t
inspire me either, so I glossed over it for years.
Converts a hash into a set of resources and adds them to the catalog.

40. I already said something about Frank’s Red Hot, right?
Almost everywhere I use it looks like this
40
$thing = loadYaml(“${yamlDir}/thing/${environment}.yaml”)
create_resources( some::define, $thing )
That same YAML gets used in a bunch of those.
backstopsolutions:
lastOctet: 226
clientFqdn: 'www.backstopsolutions.com'
seconddn: 'backstopsolutions.com'
includeFile: 'true'

41. ‣ One define uses the lastOctet and some
knowledge about our systems to set up an
interface on the load balancers
‣ Another uses the hostnames to set up a couple
of vhosts for the Rails app
‣ Yet another adds it into several DNS zones
41

42. ‣ Another benefit: it’s YAML. You don’t have to
write it by hand
‣ Grab some external data source, frob as
needed, dump out some YAML. As long as you
meet the spec, you’ll be fine
42

43. Huge gotcha here
‣ Everything you pass that YAML to needs to do
something with all the parameters named in the file
‣ Adding a field thus means changing all those
defines
‣ We have a lot of ‘$foo = “dontcare”’ in there.
Just come up with an idiom and stick to it
‣ You could probably do something clever to filter
out params?
43

44. " Maybe not everything goes
in Puppet
‣ Last year I had this slide (included verbatim)
44

45. Well, I can’t always be right
45

46. ‣ Think to yourself: do I ever want to do this thing
without running Puppet?
‣ Does it make sense for this to go through the
same test -> prod cycle?
‣ Do the same people need to see this content as
the Puppet manifests?
‣ Can it be generated in one place, or do you
need to gather data from several?
46

47. ‣ As we’ve grown to more sites and more Puppet
masters, this has hurt all over the place
‣ The deploy concept (vs `svn up`) made a mess
of the checkin of this derived data
‣ I still think it’s an OK approach, but it needs
thought
47

48. Examples
‣ DNS data: there’s only one actual copy of your
DNS, and that’s what the world can see. What
does a beta machine do with it?
‣ Load balancer configs: multiple datacenters
need to know about each other, but can’t see the
other’s database to derive config from
48

49. </TechnicalContent>
49

50. Want to help out?
We’re hiring
50
‣ Sysadmins
‣ Developers (Rails or Java)
‣ Testers
‣ Lots more
backstopsolutions.com/careers

51. DevOpsDays Chicago!
‣ October 7th and 8th @ the Sears Tower
‣ CFP open until the 22nd
‣ Registration open now! Use code
PUPPET_CAMP for 10% off
devopsdays.org/events/2014-chicago/registration/
‣ I might be on the committee
51
I refuse to call it the Willis Tower, y’all

52. Puppet Users Group
‣ Hosted by this guy, downtown, ~monthly
(coming soon!)
‣ meetup.com/Chicago-Puppet-users-group/
52

53. That’s it
Questions?
53