This is story of our journey from SaltStack to Puppet and beyond. This talk will answer following questions: - why we moved from SaltStack - why Puppet was chosen - how to use Puppet OpenSource in painless way - which orchestration tool to use with Puppet - what is next

1. From SaltStack to
Puppet and beyond...
Yury Bushmelev
SP Digital

2. What is the scale?
• Over 100 developers
• A lot of products
• Just 6 ppl in Digital Infra team at the moment
• About 200 VMs in Azure cloud

3. What's wrong with Salt?
From teamwork/long-term support perspective:
• It's feels inconsistent
• It's error prone and hard to debug
• It's YAML
• It's Jinja2 on top of YAML
• It's impossible to lint
• It's impossible to unit-test

4. Why Puppet?
• I'll explain during my talk :-D

5. Why Puppet?
From teamwork/long-term support perspective:
• Agent
• Domain Specific Language (declarative!)
• Lint/Syntax checks
• RSpec for unit tests, ServerSpec for acceptance tests
• Hiera (hierarchical database, auto-lookup)
• Ruby ecosystem (gems!)

6. Why OpenSource
Puppet?
Because we can!

7. But I heard/experienced...
• Puppet is complex/hard to learn/hard to use
• Infra code development is slower with Puppet
• ... ?

8. Puppet

9. Puppet is about the
state!

10. Puppet is about the state
• The code you write in Puppet DSL (your manifests) is
compiled by Puppet master into the catalog
• Catalog is a document describing the desired state of
every managed resource on a node
• Agent on a node will call its providers to bring the node
to desired state according to catalog records
https://puppet.com/docs/puppet/latest/subsystem_catalog_compilation.html#the-catalog-compilation-process

11. Puppet is about the state
Again:
• It's NOT sequence of steps to bring system to the state!
• It IS the state!

12. Puppet is about the state
That's why e.g.:
• You cannot easily move/rename file on a node's
filesystem if you're not managing it
• You cannot easily read a file from a node to use it in your
manifest*
• You cannot just execute a binary on a node then do
something depending on the result*
* Though you may use facts for this

13. Puppet is about the state
• You must take care when renaming/moving things:
• Ensure old resource is absent
• Ensure new resource is present
• It's usually good to keep both in the state for a while

14. Puppet is about the state of
a node
• You cannot easily do changes on multiple nodes in
defined order (do Kafka cluster rolling upgrade e.g.) using
just Puppet*
* Though you may do it using orchestration tool!

15. How to use Puppet in
painless way

16. VCS & CI/CD FTW!

17. Control repo
[Git] repository where your Puppet manifests
resides...
... and hiera data
... and unit/integration/acceptance tests
... and more

18. Control repo examples
• Puppetlab: https://github.com/puppetlabs/control-repo
• Azalio: https://github.com/azalio/control-repo
• PSICK: https://github.com/example42/psick

19. Roles & Profiles
• Role is... just VM "role" (e.g. PostgreSQL server, Puppet
master, Kafka broker, ...)
• Node can have only one role
• Role may contain multiple profiles
• Role is not configurable (cannot take any parameters)
• Profile is class describing single logical unit (e.g. docker,
ssh, consul agent). Profile is configurable (can take
parameters from Hiera e.g.).
https://puppet.com/docs/pe/latest/the_roles_and_profiles_method.html

20. Profiles
• profile::common::packages
• profile::prometheus::node_exporter
• profile::puppet::agent

21. But how to attach the role
to the server?
• Based on hostnames - do not use it (do not encode
metadata in your hostnames)
• Based on trusted facts (DO NOT use non-trusted facts for
this)
• Using External Node Classifier (ENC)
• Hiera may be used as ENC

22. –Puppet documentation
“Hiera is a built-in key-value configuration data
lookup system, used for separating data from
Puppet code.”
Hiera

23. Hiera---
version: 5
defaults: # Used for any hierarchy level that omits these keys.
datadir: data # This path is relative to hiera.yaml's directory.
data_hash: yaml_data # Use the built-in YAML backend.
hierarchy:
- name: "Per-node data" # Human-readable name.
path: "nodes/%{trusted.certname}.yaml" # File path, relative to datadir.
- name: "Per-datacenter business group data" # Uses custom facts.
path: "location/%{facts.whereami}/%{facts.group}.yaml"
- name: "Global business group data"
path: "groups/%{facts.group}.yaml"
- name: "Per-OS defaults"
path: "os/%{facts.os.family}.yaml"
- name: "Common data"
path: "common.yaml"

24. Hiera
data/common.yaml
data/product
data/product/devops_prod.yaml
data/product/devops_qa.yaml
data/os/RedHat-7.yaml
data/os/Ubuntu-16.04.yaml
data/os/Ubuntu-18.04.yaml
data/node/puppetmaster.tld.yaml
hiera.yaml
- name: "Yaml backend"
data_hash: yaml_data
paths:
- "node/%{trusted.certname}.yaml"
- "product/%{product}_%{env}.yaml"
- "os/%{facts.os.name}-%{facts.os.release.major}.yaml"
- "os/%{facts.os.name}.yaml"
- "common.yaml"

25. Hiera automatic parameters
lookup
data/common.yaml
profile ::container ::docker ::ensure: present
profile ::container ::docker ::storage_driver: 'overlay2'
profile ::container ::docker ::log_opts:
'max-size': '10m'
/data/node/podmantest.tld.yaml
profile ::container ::docker ::ensure: absent
site-modules/profile/manifest/container/docker.pp
# Install and configure docker
class profile ::container ::docker (
Enum['absent', 'present'] $ensure = 'present',
Optional[String] $storage_driver = undef,
Optional[Hash] $log_opts = undef,
...
) { ... }

26. CI/CD steps
• Checkout the control repo (usually done by your CI/CD
software)
• Run syntax and style checks
• Run unit tests/integration tests/acceptance tests
• Deploy manifests and modules onto every Puppet server
you have

27. Puppet Development Kit
• PDK provides integrated testing tools and a command
line interface to help you develop, validate, and test
modules.
pdk new module <module_name>
pdk validate
pdk test unit
• https://puppet.com/docs/pdk/1.x/pdk.html

28. Puppet-lint
• Check that your Puppet manifests conform to the style
guide
• https://github.com/rodjek/puppet-lint

29. Puppet-syntax
• Syntax checks for Puppet manifests and templates
•puppet parser validate **/*.pp
• https://github.com/voxpupuli/puppet-syntax

30. RSpec-puppet
• RSpec tests for your Puppet manifests
• https://github.com/rodjek/rspec-puppet/

31. Onceover
• Onceover is a tool to automatically run basic tests on an
entire Puppet controlrepo.
• https://github.com/dylanratcliffe/onceover

32. Puppet-litmus
• Providing a simple command line tool for puppet content
creators, to enable simple and complex test deployments
• Puppet acceptance tests using ServerSpec in other
words. Vagrant and Docker provisioners are supported.
$ pdk bundle exec rake 'litmus:provision[docker, centos:7]'
$ pdk bundle exec rake litmus:install_agent
$ pdk bundle exec rake litmus:install_module
$ pdk bundle exec rake litmus:acceptance:parallel
$ pdk bundle exec rake litmus:tear_down
• https://github.com/puppetlabs/puppet_litmus/

33. What to deploy
• main manifests directory
• site modules (roles & profiles)
• hiera config & data
• external modules
• environment.conf
• [environment version script]

34. How to deploy
• Old'n'good scp/rsync from CI/CD server
• Build code & modules archive by CI/CD pipeline and
upload to an artefacts storage. Then notify masters to
fetch and deploy it. r10k tool may be useful here.
• Any way you'd like actually. Environment is just directory
on Puppet server.

35. Workflow example 1/3
•git checkout -b feature_branch
• <edit files>
• <run local checks>
•git push --set-upstream origin feature_branch
• CI/CD server run lint/syntax checks and unit/integration/
acceptance tests
• CI/CD server deploy feature_branch on Puppet server as
'feature_branch' environment

36. Workflow example 2/3
• Go to the server your changes are intended for
•puppet agent --test --environment
feature_branch --noop
•puppet agent --test --environment
feature_branch

37. Workflow example 3/3
• Make pull request/merge request into your control repo
• Ask your teammates for review
• Merge your changes to production branch and allow CI/
CD pipeline to deploy it to Puppet server
• Wait some time for changes to be applied across the fleet
• Enjoy :)

38. Secrets

39. How to keep your secrets
• show_diff => false
• Sensitive type
• Deferred calls (using Hachicorp Vault e.g.)
• node_encrypt module (https://forge.puppet.com/
binford2k/node_encrypt)
• eyaml Hiera backend
• Hashicorp Vault Hiera backend

40. Orchestration

41. Orchestration
• Ansible :-D
• Puppet Bolt (https://puppet.com/docs/bolt/)
• MCollective (obsoleted)
• Choria (https://choria.io/)

42. Choria
Orchestration framework

43. Choria
•mco facts os.distro.description
•mco rpc service status service=puppetserver -I /
puppetserver[0-9].tld/
•mco filemgr --file /etc/puppetlabs/puppet/puppet.conf status -C
role ::default -F trusted.extensions.pp_product=devops

44. Choria
$ mco facts kernelrelease -F trusted.extensions.pp_environment=qa
Report for fact: kernelrelease
4.15.0-1066-azure found 1 times
4.15.0-1069-azure found 3 times
4.15.0-1071-azure found 31 times
Finished processing 35 / 35 hosts in 37.75 ms

45. Choria playbooks
plan spd ::opentsdb ::restart (
String $product,
Enum['qa', 'prod'] $env,
String $role,
Boolean $reboot = false,
Boolean $silent = false,
Boolean $noop = false,
) {
# Discover opentsdb cluster nodes
$nodes = choria ::discover(
discovery_method => 'choria',
classes => [ $role ],
facts => [
"trusted.extensions.pp_product=${product}",
"trusted.extensions.pp_environment=${env}",
],
test => true,
)
# Check cluster state
unless choria ::run_playbook('spd ::opentsdb ::check_health',
targets => $nodes,
silent => $silent,
) {
fail('Cluster is not healthy!')
}
https://choria.io/docs/playbooks/
$sorted_nodes = $nodes.sort
# Restart every node in order
$sorted_nodes.each |$node| {
choria ::run_playbook('spd ::util ::restart_node',
target => $node,
service_name => 'opentsdb',
silent => $silent,
reboot => $reboot,
noop => $noop,
)
# Check cluster state
unless choria ::run_playbook('spd ::opentsdb ::check_health',
targets => $sorted_nodes,
silent => $silent,
) {
fail("Node ${node} is failed! Please check cluster health!")
}
# Temporary safeguard
unless $noop {
reboot ::sleep(30)
}
}
$sorted_nodes
}

46. Choria playbooks
$ mco playbook run spd ::opentsdb ::restart --modulepath /etc/
puppetlabs/code/environments/production/modules --product abc --env
qa --role role ::opentsdb --silent --reboot

47. In-house development
• PDK-managed control repo with useful Rake tasks
• Vault PKI certificates management tool (can check cert expiration
and issue new cert in PEM and PKCS12 formats)
• Choria 'query' agent (can query prometheus exporter and REST url
for data in a playbook)
• Choria Kafka/Cassandra/Elasticsearch/OpenTSDB/RabbitMQ*
rolling reboot playbooks
• Kafka topic management Puppet provider*
* under development now

48. What's beyond?

49. Immutable
infrastructure

50. Immutable infra in short
words
• Manage VM images instead of VMs
• Recreate whole VM when things are changed

51. Immutable infra questions
• What to do with persistent data?
• How to customise image on boot (host-dependent
settings)?
• What about orchestration?

52. Thank you!

53. Q & A
Yury Bushmelev
Twitter: @jay7t
Github: jay7x
https://www.linkedin.com/in/yuribushmelev