1.
DECONFIGURATION
MANAGEMENT
MAKING PUPPET CLEAN UP ITS OWN MESS
JOSH SNYDER

2.
WHAT'S THE PROBLEM?
We use Puppet
We change things

3.
I have been known to deploy 20 times daily
And Puppet runs every 30 minutes

4.
That thing you deployed in a hurry isn't always in the place it should be
long term.
Or...we deploy a thing, and decide next week (or tomorrow) that it isn't
right for us
Or we just straight-up make mistakes
Or our coworkers make mistakes
git revertcan't save me

5.
My colleague sends me a PR to deploy memcached config files under a new
path. Should I comment on their failure to clean up the old ones?
YES!

6.
But they're busy
But I'm busy
And I really want to get home to make dinner

7.
WHAT ARE OUR OPTIONS?

8.
IGNORE THE PROBLEM
Ignore it! It probably won't hurt anything
Except it will confuse people.
It WILL confuse you, at two AM
JVM versions will exist on some older dev machines, but not newer ones

9.
WRITE CODE TO CLEAN UP AFTER OTHER CODE
file { '/etc/memcached.conf':
ensure => absent,
}

10.
ENSEMBLES OF RESOURCES GET COMPLEX
if $ensure == 'present' {
file { '/etc/init/memcached.conf':
ensure => 'file',
...
}
} else {
file { '/etc/init/memcached.conf':
ensure => 'absent',
...
}
}

11.
NO FUN AT ALL
more code to write
more code to review
my cleanup code might be broken
someone has to clean up the cleanup code

12.
WRITE MORE CODE (IN A DIFFERENT LANGUAGE)
Use Ansible clean up a er Puppet?

13.
REBUILD MACHINES
Good! Cattle, not pets
I'm deploying new changes every 30 minutes
We only clean things up every day?
30 days? 60 days? Five years?
∃ an impedance and tooling mismatch

14.
IMMUTABLE IMAGES
A more aggressive version of rebuilding machines
Use if it suits you

15.
THE SOLUTION (TO ALL YOUR PROBLEMS)
1. Use Puppet to specify what should be deployed
2. Allow Puppet to remove anything it doesn't know about

16.
WHAT'S TO COME?
A few basic examples (directory purging, cronjobs, hosts)
A bit of Puppet internals
Use Puppet internals to achieve more purging

17.
BATTERIES ARE INCLUDED

18.
DIRECTORY PURGING
file { '/etc/cassandra':
ensure => 'directory',
recurse => true,
purge => true,
force => true,
...
}

19.
WHAT WILL THIS DO?
Use puppet agent --noop
Or add noop => true
Look at the system:
$ ls /etc/cassandra
...
$ dpkg -S /etc/cassandra
...

20.
EXAMPLE 1: PARTIAL MANAGEMENT
Situation: We want to purge /etc/cassandra, but we need to generate
the list of seeds outside of Puppet
file { '/etc/cassandra':
ensure => directory,
recurse => true,
purge => true,
force => true,
...
}
file { '/etc/cassandra/seeds':
ensure => file,
replace => false,
...
}

21.
EXAMPLE 2: CRONJOBS
Lots of cronjobs
Lots of cronjobs!
Using Yelp's
One file per job in /etc/cron.d
puppet-cron

22.
PURGING CRONJOBS
Problem: purge => truewould wipe out crons provided by Debian
packages.

23.
Option ImplicationsOption Implications
Recompile cron to read from a
supplemental directory
Anyone else using
would have to use our patched cron
Create Fileresources for each
file we expect a from a deb.
Whenever someone installs a
package with a new cronjob in it,
they'd get a nasty surprise
Find some way to identify those
cronjobs that were originally
created by Puppet
Good
puppet-cron

24.
What this solution ends up looking like: (ish)
file { '/nail/etc/cron.d':
ensure => directory,
purge => true,
force => true,
recurse => true,
}
file { '/nail/etc/cron.d/myjob':
ensure => file,
...
} ->
file { '/etc/cron.d/myjob':
ensure => link,
target => '/nail/etc/cron.d/myjob',
}
github.com/Yelp/puppet-cron

25.
EXAMPLE 3: /ETC/HOSTS
Puppet agent has a RAL
(resource abstraction layer)
RAL is responsible for
representing resources on the
system as Puppet Resource objects
$ puppet resource host
host { 'ip6-allnodes':
ensure => 'present',
ip => 'ff02::1',
target => '/etc/hosts',
}
host { 'ip6-allrouters':
ensure => 'present',
ip => 'ff02::2',
target => '/etc/hosts',
}
host { 'localhost':
ensure => 'present',

26.
Puppet diffs resources in the catalog against the RAL it constructs
Could we ask it remove resources present in the RAL but not in the catalog?
YES!

27.
IT'S THIS SIMPLE
resources { 'host':
purge => true,
}
Use noop => trueto try before you buy

28.
HOW IT WORKS
All on the agent, a er catalog compilation
Iterate over resources, calling the generateor eval_generate
method on each.
Each resource has the opportunity to add more resources to the Puppet
run.

29.
Walkthrough: fetching files from a fileserver
file { '/etc/cassandra':
ensure => directory,
source => 'puppet:///modules/cassandra/config_dir',
recurse => true,
purge => true,
force => true,
}
1. Get catalog with this resource declared
2. Puppet agent calls eval_generateon this resource
3. eval_generateexamines the disk, compares it with the Puppet
fileserver
4. Generates more resources to represent the files beneath this directory

30.
HOW THE RESOURCESTYPE WORKS
resources { 'host':
purge => true,
}
1. Puppet calls generate
2. Generate finds all resources of type Hostin the catalog
3. Asks providers of Hostfor their instances
4. Compare the two
5. Emit new resources:
host { 'ip6-allnodes':
ensure => absent,
}

31.
MOVING BEYOND THE BUILTINS

32.
PURGING UNDESIRED DEBIAN PACKAGES
Let's say I do:
resources { 'package':
purge => true,
}
This happens:
Notice: /Stage[main]/Main/Package[libxtst6]/ensure: current_value
Notice: /Stage[main]/Main/Package[libxcb-dri3-0]/ensure: current_va
Notice: /Stage[main]/Main/Package[powermgmt-base]/ensure: current_v
Notice: /Stage[main]/Main/Package[python3-py]/ensure: current_value
Notice: /Stage[main]/Main/Package[libtk8.6]/ensure: current_value
Notice: /Stage[main]/Main/Package[node-ansi-color-table]/ensure:
Notice: /Stage[main]/Main/Package[libxpp3-java]/ensure: current_val
Notice: /Stage[main]/Main/Package[python3-newt]/ensure: current_val
Notice: /Stage[main]/Main/Package[bsdmainutils]/ensure: current_val
Notice: /Stage[main]/Main/Package[libpulse0]/ensure: current_value
Notice: /Stage[main]/Main/Package[liblvm2app2.2]/ensure: current_va

33.
NO BUENO
Why doesn't puppet understand that it should remove all packages that:
aren't in the catalog
no other package depends on

34.
We need to be dependency-aware
What removes packages and is dependency aware?
apt-get autoremove

35.
APT-GET AUTOREMOVE
Divides packages into:
manually installed (we're sure we want this)
auto installed (a dependency)

36.
THIS MAPS WELL TO PUPPET
puppet state ⇒ autoremover state
puppet state ⇒ autoremover state
in catalog manually installed
not in catalog automatically installed

37.
AN IMPLEMENTATION COMES TOGETHER
1. Synchronize the autoremover database with the Puppet catalog
2. Run apt-get autoremove
3. Problem?

38.
AN IMPLEMENTATION COMES TOGETHER (PARTE DUEX)
1. Synchronize the autoremover database with the Puppet catalog
2. Run apt-get -s autoremove
3. Read the output and create Puppet package resources
4. Much rejoicing!
github.com/hashbrowncipher/puppet-package_purging

39.
GENERAL PURPOSE SOLUTIONS?
Could there be a jack-of-all-trades solution to purging?
What if we could do:
purge { 'user':
unless => [ 'uid', '<=', '500' ],
}
It exists: github.com/crayfishx/puppet-purge

40.
BONUS FUN STUFF

41.
NOT MY FAVORITE DEFAULT
Q: What does this do?
package { 'mysql-server-5.7': }
package { 'bash': }
A: Creates version dri

42.
UPGRADE (SOME OF THE) THINGS
package { 'mysql-server-5.7':
ensure => $my_favorite_mysql_version
}
package { 'bash': }
aptly_purgecan set all versioned packages as held by dpkg
aptly_purge { 'packages':
hold => true,
}
Upshot: apt-get dist-upgradeand unattended-upgrades
will only touch packages without a specific Puppet version specified.

43.
END MATTER
Please do tell me your stories of Puppet resource purging.
Walk up and say hi right now.
works toojosh@code406.com
in Puppet Community Slack@josnyder