A presentation given at PuppetCamp Chicago 2015 on the Puppet Development Workflow used at Grubhub. It walks through philosophy as well as process.

1. Puppet Development Workflow

2. Bio - Jeff Smith
• Manager, Site Reliability Engineering
at Grubhub
• Puppet User for about 2 years
• Yes, we are also hiring.
• Yes, there is free food. Yes, it's totally
awesome to work here.
Email: jsmith@grubhub.com
Twitter: @DarkAndNerdy
Blog: http://www.allthingsdork.com

3. Agenda
• High level Environment Overview
• Local workstation setup
• How we solve problems
• Branching strategy
• Committing, Automated Testing and Publishing

4. Puppet Environment

5. Local Workstation Setup
Editor Setup
• VIM
• tmux
• Nerd Tree
• Powerline

7. Local Testing Tools
Listing tools can help you can syntax errors before you commit
code!
Local Linting Tools
• jsonlint
• puppet parser validate
• erb syntax checker

8. VIM Linting Function Example
function LintFile()
let l:currentfile = expand('%:p')
if &ft == 'puppet'
let l:command = "puppet-lint " . l:currentfile
elseif &ft == 'eruby.html'
let l:command = "erb -P -x -T '-' " . l:currentfile . "| ruby -c"
elseif &ft == 'json'
let l:command = 'jsonlint -q ' . l:currentfile
end
silent !clear
execute "!" . l:command . " " . bufname("%")
endfunction
map :call LintFile()

9. Guard
Guard is a command line tool to easily handle events on file
system modifications.
guard-puppet - A configuration of Guard geared towards
Puppet manifests and syntax.
Find more at https://github.com/johnbintz/guard-puppet

10. Vagrant Environment
It's easier to make changes locally then to a committed
repository. Use Vagrant for test VMs
• Puppet Master
• 2 Agents (only 1 started by default)
• Shares the puppet repo on the Host machine and uses that
as the module path in the VM

11. config.vm.define "master" do |master|
master.vm.box = "centos65-base-small"
if not ENV['PUPPETREPO'].nil?
puppet_path = ENV['PUPPETREPO']
else
puppet_path = ENV['HOME']+'/Development/puppet'
end
config.vm.synced_folder puppet_path, "/manifests", type: "nfs"
master.vm.network "private_network", ip: "172.16.1.10"
master.vm.network :forwarded_port, host: 10443, guest: 443
config.vm.hostname = "pe.local.vm"
master.vm.provision :hosts
config.pe_build.version = '3.3.2'
config.pe_build.download_root = 'http://s3.amazonaws.com/specialbuildbucket/'
master.vm.provision :hosts do |provisioner|
provisioner.add_host '172.16.1.11', ['agent.local.vm', 'agent']
provisioner.add_host '172.16.1.12', ['agent-2.local.vm', 'agent-2']
end
master.vm.provision :pe_bootstrap do |provisioner|
provisioner.role = :master
#provisioner.relocate_manifests = true
end
master.vm.provision :shell, inline: "service iptables stop;chkconfig --level 3 iptables off"
master.vm.provision :shell, path: 'bootstrap.sh'
end

12. How We Approach Development
Guiding Principles
• YAGNI - You Ain't Gonna Need It
• DRY - Don't Repeat Yourself
• Modules, Profiles, Roles
• Have respect for what you're doing
• Write your manifests in Puppet

13. You Ain't Gonna Need It
It's great to build your module to support CentOS, Ubuntu,
Solaris, AIX. But if your shop only has CentOS, has only ever
used CentOS and has plans to only use CentOS...........code for
CentOS.
• Adds too much time to development
• Complicates a potentially simple solution
• Not being tested in those environments so probably won't
work anyways

14. Don't Repeat Yourself
When you see the same piece of code showing up over and
over again, there's an opportunity.
• Limit the scope of where changes need to happen
• Compose your modules in a way that makes them reusable
• Hard-Coding is the devil

15. Modules, Profiles and Roles
Compose your Puppet manifests into 3 separate categories
• Modules - Contains the implementation details of a module.
Responsible for the plumbing.
• Profiles - Leverages modules to implement business logic.
Takes your "Apache" module and turns it into "Payroll_Web"
• Roles - Takes all of the profiles necessary to make a
complete system.
Only assign roles to a node and only 1 role per node

16. Have Respect for What You're Doing
YOU ARE PROGRAMMING

17. Write Your Manifests in Puppet
file { '/opt/app/config.ini':
ensure => file,
content => template('config.erb')
}
OR
payroll_config { 'database_connection':
property => 'dbserver',
value => 'dbserver1.example.com:3306'
}

18. Time to Write Some Code!

19. Puppet Forge is
Your Friend
• Someone has the same problem as
you
• More eyeballs on the same problem
• More flexible
• Less Work

20. Branching Strategy
Largely will depend on your environment. We have a single
Puppet repository for everything. (Don't do this)
3 Phases of Development
Local => Preprod => Production

21. Local Development
• Use Vagrant for test VMs
• Make all changes prior to commit
• Use linting
• Test using your local Puppet Master VM and Agent
• Set FACTS or variables via Custom Facts

22. Writing Test Cases
Be highly selective about what you test in Puppet.
Puppet Code
file { '/opt/app/config.ini':
ensure => file,
content => template('app/config.erb')
}
RSpec Puppet Test
it { should contain_file('/opt/app/config.ini) }

23. Puppet

24. Things To Test
1. File Syntax
2. Conditional Branches
3. Interpreted Values (e.g. RegEx evaluations, Facts)
4. Catalog Compilation

25. Getting Code to the Puppet
Master
• Work off a branch. Branch name should match your ticket
• Push your branch to the remote origin server
• Push your branch to the Pre-prod Puppet Master Remote
Repo (optional)
• Create a Pull Request from your branch, into the Develop
Branch

26. Branching Workflow Diagram

27. Automated Test Execution
CI Server watches master/develop branch. Executes tests on
change
1. Script executes to determine which files have changed
2. Linting is performed on changed files (if applicable)
3. Catalog compilation of changed manifests
4. Execution of manifests specific tests (if applicable)
5. Deployment of code to the Puppet Masters

28. Auto Deployment

29. Building Confidence
With enough confidence in the process you should be able to
• Deploy your changes to production after a commit and
successful automated testing
• Regular Puppet runs on production systems
• Iterate on changes faster
Move slowly. Go piece by piece. Every small step you take
adds value immediately. Bite off small bits.

30. Questions?