Slides from a talk I gave at PuppetConf 2015. Abstract: I joined Constant Contact in the Spring of 2014 to help transform their Puppet infrastructure. Constant Contact was a very early adopter of Puppet and had a hard time keeping up with changes to the language. When I got to Constant Contact we were stuck on a very old version of Puppet 2.7 because our code was heavily dependent on inheritance and dynamic scoping. There was no separation of data and code and 99% of the Puppet modules in use in the environment were homegrown. With over 267,000 lines of ancient code, I was completely overwhelmed with how to get us up to speed. This talk is about how we managed to accomplish this incredible feat in just over a year.

1. 2 0 0 , 0 0 0 L I N E S L AT E R
O U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E
D AV I D D A N Z I L I O
C O N S TA N T C O N TA C T

2. W H O A M I ?
• Puppet Evangelist at
Constant Contact
• Joined Constant Contact
in 2014
• SysAdmin background but
more of a developer these
days
• Second time speaking at
PuppetConf

4. W H AT I S C O N S TA N T C O N TA C T ?
B A C K G R O U N D

5. • Founded in 1995
• Headquarters in Waltham, MA
• Offices in San Francisco, NYC, London, among others
• ~ 1,500 Employees, ~ 350 developers, ~ 70 operations
• 160+ apps (SOA - microservices for hipsters)
• We’re hiring! (See me after for details)

6. W H AT ’ S T H I S TA L K A B O U T ?
• An overview of our environment and our challenges
• Our successes, failures, and lessons from trying to
transform Puppet in a large enterprise
• Plans for achieving the Puppet singularity
• My quest to understand and affect change in a
massively complex organization

7. R U N N I N G P U P P E T AT
S C A L E I S H A R D
T L ; D R

8. A L O N G T I M E A G O
I N A G A L A X Y FA R FA R A WAY

9. E A R LY A D O P T E R S Y N D R O M E
• Started using Puppet in 2009 with version 0.24.8
• You name the bug, we’ve probably seen it
• Forge didn’t exist so we had to write everything
• Puppet talent was (still is) hard to come by
• Writing Puppet was a little harder back then…

10. Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.x
Dynamic Scope X X X X X
Appending to attributes in class inheritance (+>) X X X X X X X X
Multi-line C-style comments X X X X X X X
Arrays of resource references allowed in relationships X X X X X X X
Overrides in class inheritance X X X X X X X
Appending to variables in child scopes (+=) X X X X X X X
Class names starting with 0-9 X X X
Regular expressions in node definitions X X X X X X
Assigning expressions to variables X X X X X X
Regular expressions in conditionals/expresions X X X X X X
elsif in if statements X X X X X
Chaining Resources X X X X X
Hashes X X X X X
Class Parameters X X X X X
Run Stages X X X X X
The “in” operator X X X X X
$title, $name, and $module_name available in parameter lists X X X X X
Optional trailing comma in parameter lists X X X X
Hyphens/dashes allowed in variable names * X
Automatic class parameter lookup via data bindings X X X
“Unless” conditionals X X X
Iteration over arrays and hashes X X
The modulo (%) operator X X
$trusted hash X

11. • Lots of the code we wrote back then is still around…
• The company was growing so fast, nobody had time to
focus on Puppet
• Ops mandated that all configuration and app
deployment had to be done with Puppet
• Lots of push back
• Puppet became a dirty word

12. T H E G O O D
• Everything is in Puppet
• Everything is in Git
• Our infrastructure is built to be ephemeral
• We can provision physical nodes about as fast as we
can spin up a virtual machine
• Moving an app to OpenStack is just a matter of running
Puppet

13. T H E B A D
• Stuck on Puppet 2.7.3; lots of the code was written
before parameterized classes
• Most of our code lives in a single Git repo
• Testing is up to the developer
• No design beyond the initial implementation; most of
the code just evolved organically
• Propensity for home-grown solutions, and a habit of
thinking only in terms of the current use case

14. T H E U G LY
• Massive codebase (more on this in a second)
• Frightening complexity
• Tightly coupled codebase (massive understatement)
• Very few tests (effectively none)
• Very little documentation
• Too easy to get into production
• No dependency management

16. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 1
1 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4
3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 2
1 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 0
5 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6
5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 8
5 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 8
6 7 S Q L 4 5 2 1 9 7 9 , 9 9 9
4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 9
1 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8
1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5

17. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4
4 0 E R B 6 1 7 1 1 2 , 2 7 1
7 X M L 5 0 2 1 , 5 5 4
5 6 YA M L 1 6 4 1 , 5 1 1
3 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 8
1 6 J S O N 7 5 0 1 , 1 7 5
8 2 P U P P E T 1 5 9 4 6 6 9 4
2 3 S M A R T Y 3 8 0 3 6 9
2 L I S P 4 4 9 6 3 0 0
1 4 O T H E R 1 3 3 1 1 3 6 2 9
2 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5

18. T H E J O U R N E Y B E G I N S

19. U N D E R S TA N D

20. I D E N T I F Y O U R U S E R S
• Consumers use Puppet to deploy their apps. They just
want Puppet to work.
• Developers may write Puppet modules or profiles.
They may consume Forge modules.
• Architects help shape the strategic direction of Puppet
in the organization.

21. • Consumers have a very different use case than
Architects or Developers.
• Most of our users are Consumers, so we should focus
on them.
• Consumers aren’t necessarily interested in learning
Puppet, but they need to use it to deploy their apps.
• Interfaces are very important to these people.

22. T E A C H

23. T E A C H I N G P U P P E T
• Most people working with Puppet didn’t really know the
language well
• But they had to get shit done
• Rampant propagation of bad patterns
• Wrote a 4-hour intro class and started teaching
• Get everybody on a common baseline, a good
foundation to build good habits
• Offer more advanced training to people who want it

24. U P G R A D E

25. A M O D E R N V E R S I O N O F P U P P E T
• Puppet 2.7.3 is buggy and slow
• Couldn’t use any modules from the Forge out of the
box because of trailing commas
• Puppet 2.7 was holding us back
• Offload file serving to Puppet 3 masters, cheap
performance win: bit.ly/1VfSwYM
• Problem: code isn’t compatible with Puppet 3

26. class mysql_site {
# Set some defaults
$mysql_log_rotate_enable=0
$mysql_log_rotation='daily'
$mysql_log_retention='14'
$mysql_slowlog_rotate_enable=0
$mysql_slowlog_rotation='weekly'
$mysql_slowlog_retention='4'
$mysql_errlog_rotate_enable=0
$mysql_errlog_rotation='monthly'
$mysql_errlog_retention='6'
$mysql_genlog_rotate_enable=0
$mysql_genlog_rotation='daily'
$mysql_genlog_retention='14'
...AND SO ON…
}
class mysql_site::dev inherits mysql_site {
# Overrides for Dev environment
$mysql_log_rotate_enable=1
$mysql_slowlog_rotate_enable=1
$mysql_errlog_rotate_enable=1
$mysql_genlog_rotate_enable=1
}
class mysql_site::dev::app
inherits mysql_site::dev {
# Overrides for app in dev environment
$mysql_slowlog_rotation='monthly'
$mysql_errlog_rotation='daily'
include mysql
}

27. M I N I M U M
V I A B L E R E FA C T O R

28. • We knew the code was bad, but we didn’t have the
resources to focus on a redesign
• Refactor just enough to get to Puppet 3
• Fix templates
• Eliminate dynamic scope
• Hiera played a big role in this
• Move hosts to Puppet 3 as code becomes compatible

29. TA K I N G T H E P L U N G E

30. • We built out a Puppet 3 infrastructure alongside the
legacy infrastructure
• Shared CA, so the clients could talk to both sets of
masters
• Move hosts over as the code becomes compatible
• Turns out this is really hard when there’s no standard
way to classify hosts

31. • We started with the easiest stuff: our own hosts
• Moved on to our CD apps
• In process of moving legacy apps now

32. M E A N W H I L E …

33. • We couldn’t stop time and do all this work in a vacuum,
life was still happening around us
• New modules are put into their own repos and
deployed with r10k
• We’ve kept Puppet 3 environment up to date. Started
on 3.4.3, now on 3.8.3
• New modules must have tests and documentation
• We’re using roles and profiles for new things

34. • CI workflow and infrastructure
• Testing environments, harnesses, and helpers
• Deployment tooling
• Knowledge of testing frameworks
• No official images

35. N E X T S T E P S

36. T H I N G S W E ’ R E TA C K L I N G N E X T
• Node classification
• Roles and Profiles
• Decoupling our code
• Desire for bespoke everything
• Writing modules with a single entry point
• Dependency management
• The Forge test

37. T H E F O R G E T E S T / P U P P E T A P P R O V E D
T E S T / D E F I N I T I O N O F D O N E
• Can this module be deployed to the Forge?
• Is there any proprietary data embedded?
• Is it modular enough to meet unforeseen use cases?
• Does it have a robust suite of tests?
• Does it conform to community style expectations?
• Is it fully documented?

38. I T ’ S A L L A B O U T T H AT I N T E R FA C E
• Make it easier to use our modules
• Our consumers care about the interface, not the
implementation
• Design better and fewer interfaces
• Modules with well defined interfaces for composability
and better dependency management
• Documentation and tests as part of the definition of
done

39. T E S T I N G
• Make testing easier
• Make testing automatic
• Make testing a part of the culture
• Teach people about BDD
• Show them how much easier and more reliable an
automated test can be than a manual one
• Eventually, make testing required

40. L E S S O N S L E A R N E D

41. • Know your audience
• Don’t try and please everybody
• It’s always going to take longer than you think
• Don’t forget about the supporting infrastructure
• People, process, and technology. In that order
• It’s never going to move as fast as you want it to. Walk,
don’t run

42. • Don’t let yourself become paralyzed by complexity
• Take notes, write down things you need to work on,
you’ll get to them eventually
• Always take the easy wins
• Focus on high impact improvements, help as many
people as you can
• Small improvements add up

43. There’s nothing more successful than success

44. L E S S O N S T O B E L E A R N E D

45. • How do we get people to feel invested in the design?
• How can we reenforce good behaviors?
• How can we make sure our massive codebase is being
regularly evaluated?
• How can (should?) we hold people accountable for
their Puppet code?
• How can we get operations folks to realize they’re
software developers?

46. T H E E N D

47. H O W T O F I N D M E
• @djdanzilio on Twitter
• danzilio on Freenode (you can usually find me in
#puppet-community, #puppet-dev, and #puppet)
• danzilio on GitHub and The Forge
• david (at) danzil (dot) io
• blog.danzilio.net

48. I M A G E C R E D I T S
• The space photos are from the NASA Image Galleries
and are either in the public domain or licensed under
CC-BY 2.0
• All other photos are mine or my employer’s