Orchestrated Functional Testing with Puppet-spec and Mspectator - Raphaël Pinson, Camptocamp

1. ORCHESTRATED FUNCTIONAL
TESTING WITH PUPPET-SPEC
AND MSPECTATOR
RAPHAËL PINSON

2. Who am I?
Raphaël Pinson (@raphink)
■ Infrastructure Developer & Trainer @ Camptocamp
■ Augeas & Augeasproviders developer
■ Various contributions to Puppet & ecosystem
www.camptocamp.com / 2/32

3. Monitoring vs. Functional Tests
Complementary or redundant?
www.camptocamp.com / 3/32

4. Conformity Tests
■ Check if machines comply to standards
■ Avoid permanent heavy monitoring checks
■ Tests must be inter-dependent
■ Focus on getting sysadmins to fix one thing at a time to converge
toward standards
www.camptocamp.com / 4/32

5. Treetester
■ Back in 2008
■ Written in Perl
■ Orchestrate conformity tests on a 4k+ server fleet
www.camptocamp.com / 5/32

6. Treetester: modules output
■ For all hosts/modules
■ Number of hosts filtered per module
■ Modules dependency tree
■ Colors by priority
www.camptocamp.com / 6/32

7. Treetester: host output
■ For each host
■ Failed steps in the module tree
■ Green: OK, Red: KO, Purple: Ignored
www.camptocamp.com / 7/32

8. Treetester architecture
■ All data in a database (MySQL)
■ Tests scripts output YAML
■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http)
■ Tests are inter-dependent
■ Generate filtered data as a tree
■ Generate graphs (graphviz)
www.camptocamp.com / 8/32

9. Treetester filters
■ For each test/module
■ Based on data in MySQL (joins and additional SQL conditions)
■ Allows to link tests to each other
■ Like multiple sieves
www.camptocamp.com / 9/32

10. Treetester: future?
■ Not open-sourced :'-(
■ Too monolithic/not flexible enough
■ Heavily linked to specific architecture
■ Needed a rewrite
www.camptocamp.com / 10/32

11. Adding specs to Puppet runs
■ Testing the catalog before it gets applied
■ Testing the node after the catalog is applied
Enter the Puppet-spec module
www.camptocamp.com / 11/32

12. Rspec-puppet
■ http://rspec-puppet.com
■ Now the standard to unit test Puppet manifests
■ Generates catalogs in clean environments
■ Asserts catalogs for resources/classes
require 'spec_helper'
describe 'logrotate::rule' do
let(:title) { 'nginx' }
it { should compile.with_all_deps }
it { should contain_class('logrotate::setup') }
end
www.camptocamp.com / 12/32

13. Puppet-spec
■ Runs tests from within Puppet runs
■ Test catalogs using rspec-puppet
■ Test hosts using serverspec
www.camptocamp.com / 13/32

14. Puppet-spec: Unit testing
■ Catalog exposed by PuppetSpec::Catalog.instance.catalog
■ Uses rspec-puppet matchers
■ Asserts real catalogs
■ Runs on the master or agent side (as catalog indirection terminii)
describe 'puppet' do
subject { PuppetSpec::Catalog.instance.catalog }
it { should contain_package('puppet') }
it { should contain_package('ppet') }
it { should include_class('puppet') }
it { should include_class('puppet::client::base') }
end
www.camptocamp.com / 14/32

15. Puppet-spec: Unit tests output
# puppet agent -t
info: Retrieving plugin
err: Could not retrieve catalog from remote server: Unit tests failed:
F..
Failures:
1) package
Failure/Error: it { should contain_package('augeas') }
expected that the catalogue would contain Package[augeas]
# /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3
# /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find'
Finished in 0.00092 seconds
3 examples, 1 failure
Failed examples:
rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package
info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013
notice: Using cached catalog
www.camptocamp.com / 15/32

16. Puppet-spec: Deploying unit tests
■ On the master side:
○ Tests are located in the spec/catalog/class directory of the
environment
○ Only the directories named after classes declared in the catalog
are tested
■ On the agent side:
○ Deploy tests using pluginsync
○ Tests are located in the lib/spec/catalog/class directory of each
module
○ Only the directories named after classes declared in the catalog
are tested
www.camptocamp.com / 16/32

17. Puppet-spec: Unit tests limits
■ When to apply the tests (currently based on class names)
■ Tests on master, or need to deploy all tests with pluginsync
■ Redundant with existing unit tests, or additional security?
www.camptocamp.com / 17/32

18. Puppet-spec: Setting up Unit testing
■ Tests achieved from catalog indirection terminii
■ Plugins (terminii) deployed with pluginsync
■ Setup done in routes.yaml:
agent:
catalog:
# Either on the agent side
terminus: rest_spec
cache: yaml
master:
catalog:
# Or on the master side
terminus: compiler_spec
www.camptocamp.com / 18/32

19. Serverspec
■ http://serverspec.org
■ Provides RSpec matchers for local functional tests (packages,
users, services, ports, etc.)
■ Independant from configuration management tools
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 19/32

20. Serverspec backends
Allows to use various means of launching tests:
■ SSH (default)
■ Exec
■ Puppet (RAL, removed from core)
$ serverspec-init
Select OS type:
1) UN*X
2) Windows
Select number: 1
Select a backend type:
1) SSH
2) Exec (local)
Select number: 1
www.camptocamp.com / 20/32

21. Puppet-spec: Functional testing
■ Uses serverspec/specinfra matchers
■ Tests the machine state (not the catalog)
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 21/32

22. Puppet-spec: Function tests output
# puppet agent -t
info: Retrieving plugin
info: Caching catalog for foo.example.com
info: Applying configuration version 'raphink/a2c8e0f [+]'
... Applying changes ...
notice: Finished catalog run in 59.19 seconds
err: Could not send report: Unit tests failed:
FF
Failures:
1) augeas
Failure/Error: it { should be_installed }
expected "augeas" to be installed
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
2) /usr/share/augeas/lenses/dist
Failure/Error: it { should be_file }
expected "/usr/share/augeas/lenses/dist" to be file
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
Finished in 0.06033 seconds
2 examples, 2 failures
Failed examples:
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeas
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/www.camptocamp.com / 22/32

23. Puppet-spec: Deploying functional
tests
■ Tests are run after catalog application
■ Tests can be distributed via pluginsync (in the spec/server/class)
directory of each module
■ Tests can be distributed with file Puppet resources, optionally
using the spec::serverspec defined resource type
www.camptocamp.com / 23/32

24. Puppet-spec: MCollective agent
■ Communicates with distant nodes
■ Sends action and values to specinfra check commands
■ Does not implement serverspec syntax
■ Returns true/false
■ Uses MCollective as transport (instead of SSH)
Examples:
$ mco rpc spec check action=user values=rpinson
$ mco rpc spec check action=file values=/etc/passwd
$ mco rpc spec check action=resolvable values=google.fr,A
$ mco rpc spec check action=listening values=80
$ mco rpc spec check action=process values=mcollectived
$ mco rpc spec check action=file_contain values=/etc/passwd,rpinson
www.camptocamp.com / 24/32

25. Mspectator
■ https://github.com/raphink/mspectator
■ RSpec matchers
■ Calls MCollective to achieve tests
■ Uses MCollective spec agent (among others)
www.camptocamp.com / 25/32

26. Mspectator architecture
■ Client runs RSpec
■ RSpec calls MCollective
■ MCollective calls distant spec agent
■ spec agent calls specinfra backend
www.camptocamp.com / 26/32

27. Mspectator syntax
Own matchers, mapping to specinfra backend methods:
require 'mspectator'
describe 'apache' do
it { should find_nodes(100).or_less } # Counts discovered nodes
it { should pass_puppet_spec } # Runs the `spec` agent
it { should have_certificate.signed } # Uses the `puppetca` agent
context 'when on Debian',
:facts => { :operatingsystem => 'Debian' } do # Filter by facts
it { should find_nodes(5).with_agent('spec') }
it { should have_package('apache2.2-common') }
it { should_not have_package('httpd') }
it { should have_service('apache2').with(
:ensure => 'running'
) }
it { should have_file('/etc/apache2/apache2.conf') }
it { should have_directory('/etc/apache2/conf.d') }
it { should have_user('www-data') }
end
context 'when using SSL', :classes => ['apache::ssl'] do # Filter by classes
it { should find_nodes(50).or_more }
it { should have_package('ca-certificates') }
end
end
www.camptocamp.com / 27/32

28. Mspectator output
$ rake spec SPEC=apache_spec.rb
/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb
apache
should find nodes 100
should pass puppet spec (FAILED - 1)
should have certificate
when on Debian
should find nodes 5 (FAILED - 2)
...
when using SSL
should find nodes 50 (FAILED - 3)
No request sent, we did not discover any nodes. should have package "ca-certificates"
Failures:
1) apache
Failure/Error: it { should pass_puppet_spec }
expected that all hosts would pass tests, the following didn't:
soekris01.wrk.cby.camptocamp.com:
soekris02.wrk.cby.camptocamp.com:
# ./apache_spec.rb:5
...
www.camptocamp.com / 28/32

29. Mspectator demo
www.camptocamp.com / 29/32

30. Contribute
On GitHub:
■ puppet-spec: https://github.com/raphink/puppet-spec
■ mspectator: https://github.com/raphink/mspectator
www.camptocamp.com / 30/32

31. Thank you!
■ raphael.pinson@camptocamp.com
■ @raphink on Twitter/Github
■ raphink on Freenode
■ Slides: slideshare.net/raphink
www.camptocamp.com / 31/32