Puppet Camp Paris 2015: Continuous Integration of Puppet Code (Intermediate) - François Gouteroux, D2SI

1. Continuous Integration of Puppet
Code

2. @d2si
Who am I ?
2
Jack of all trades, I love challenges of any kind from motor sports to IT.
@fgouteroux

3. @d2si
Outline
Puppet Dev and Tests
State of the art
What's missing ?
Enters beaker
Principles and common use cases
Examples
Real use case testing with beaker
Simulate a true puppet deployment
Details and Setup
Integration in the dev workflow

4. @d2si
Tests
Why ?
Syntax error è Manifest does not compile
Duplicate resource è Dependency cycle
Forgetting to include a module or set a variable è puppet run fails
Result
The host fails to enforce the expected state

5. @d2si
State of the art
Writing tests is a good way to verify that your modules are
• functional
• reusable
Ensure the module does what you want it to do

6. @d2si
State of the art
Automated testing is one of the key ways to ensure that:
• your libraries
• your manifests
are meeting your expectations !

7. @d2si
State of the art
There are differents ways to test puppet manifests
Puppet-lint
• Quick way to ensure that everybody is following a common set of conventions
• Analyze your manifests and look for deviations from the Puppet style guide

8. @d2si
Puppet-lint

9. @d2si
State of the art
Puppet-rspec
• Manifests and modules compile
• Manifests contain the expected values
• Specific types, classes or definitions are in the compiled catalog
• Parameters match the expectations

10. @d2si
Puppet-Rspec

11. @d2si
State of the art
Puppet apply
• Applies a standalone Puppet manifest to the local system
• « Simulates » a catalog compilation with modulepath option
$ puppet apply -l /tmp/manifest.log manifest.pp
$ puppet apply --modulepath=/root/dev/modules -e "include ntpd::server"
$ puppet apply --catalog catalog.json

12. @d2si
It’s cool but, beakerful
You want to be sure that running Puppet on a host will:
• build the host the way you want
• have the behavior you expect
That means, verify services like:
• SSH
• Postgres
• Nginx
are running and serving resources !

13. @d2si
What's missing ?
Testing entire systems:
1. Create virtual machines
2. Configure virtual machines
3. Deploy applications
4. Execute tests
5. Report on tests
6. Destroy virtual machines
6 5
4
3
1
2

14. @d2si
Enters Beaker
• Open source acceptance testing tool
• Built by Alice Nodelman (Puppet Labs)
• Build test environments by vm provisionning
• Multi-Cloud Providers (AWS, Google, Openstack)
• Multi Virtualisation support (Docker, vSphere, Vagrant, Virtualbox)
• Lifecycle management
• First step for Continuous Integration

15. @d2si
Principles and common use cases
• Tests are just Ruby files
• Tests passed if no errors/exceptions
• Tests use asserts to explicitly enforce a state

16. @d2si
Principles and common use cases
Let’s look at tests !

17. @d2si
Principles and common use cases
More examples !

18. @d2si
Simulate a true puppet deployment on AWS
Tasks:
1. Create 5 Virtual machines
2. Configure 4 Virtual machines
3. Deploy applications
4. Execute tests
5. Report tests
6. Destroy 5 Virtual machines

19. @d2si
TASK 1: Create VMs
How to reproduce this use case with beaker ?

20. @d2si
TASK 1: Create VMs
Create a file and describes your nodes:

21. @d2si
Beaker Roles
Each host in a host configuration file has one or more roles
Beaker natively supports the following roles:
• master
• agent
• frictionless
• dashboard
• database
These roles indicate what Puppet responsibilities the host will assume.
If puppet is installed as part of the Beaker test execution then the roles will be
honored (ie, the host defined as master will become the puppet master node).
Other than puppet installation, the roles provide shortcuts to access nodes

22. @d2si
TASK 2: Configure VMs

23. @d2si
TASK 2: Configure VMs
Step 1: Bootstrap master node
• Update package list
• Install puppetmaster, ruby packages
• Install/Configure Hiera
• Install/Configure/Deploy r10k
• Run puppet apply
Step 2: Bootstrap agent nodes
• Update package list
• Install curl, augeas packages
• Push hostname and public IP to Consul KV
• Install puppet agent
• Set Default Puppet Environment
• Run puppet agent

24. @d2si
r10k is a tool to help manage dynamic environments.
• Creates new environments from git branches
• Deploy modules to that environment.
R10K

25. @d2si

26. @d2si
beaker_helper.rb

27. @d2si
TASK 3: Deploy applications

28. @d2si
Puppet Stack – Roles/Profiles
From Craig Dunn: http://www.craigdunn.org/2012/05/239/
• Nodes get a single role
• Roles use several profiles
• Profiles and Roles are Puppet modules

29. @d2si
Puppet Stack - Hiera
From Craig Dunn
• Top down hierarchy for overriding configuration values based on
roles, environments…
• Puppet modules without hard-coded data are easily shared and
more re-usable
• Infrastructure configuration can be managed without needing to
edit Puppet code
Pluggable Backends:
• Source data from multiple locations
• Data source is abstracted from code

30. @d2si
Then it’s puppet job !
• Puppet get the node’s role with the node’s name
• Puppet apply the manifest according to the role
For front-server role, puppet need backends server infos like:
• hostname
• public IP
To get these infos, puppet call hiera consul backend.
TASK 3: Deploy applications

31. @d2si
front_server.pp

32. @d2si
apache_server.pp

33. @d2si
TASK 4: Execute Tests

34. @d2si
app_server.rb

35. @d2si
front_server.rb

36. @d2si
client.rb

37. @d2si
Task 5: Reporting

38. @d2si

39. @d2si

40. @d2si
Task 6: Destroy VMs

41. @d2si
Integration in the dev workflow
Developper update
a module
Trigger a hook
Run puppet-lint
Run puppet-rspec
Run integration tests
Merge dev to prod
(pipeline)
Run tests Play configuration

42. @d2si
Questions ?

43. @d2si
Go further
https://github.com/fgouteroux/demo-puppet-beaker
http://blog.d2-si.fr/2015/03/18/technique-puppet-beaker/

44. @d2si
Thank you !