An introduction to the Example42 NextGen Modules and their vision.

1. PUPPET MODULES:
AN HOLISTIC APPROACH
PuppetCamp Dublin 2012
Alessandro Franceschi

2. LAB 42
• 2007 - Meet Puppet. Managed the Bank of Italy webfarm
• 2008 - First generation of Lab42 Puppet Modules
• 2009 - Multi OS support and standardization of the modules
• 2010 - A redesigned and coherent Example42 Module set
Puppet Modules Standards and Interoperability (PuppetCamp Europe 2010 - Belgium)
Re-Use your Modules! (PuppetCamp 2010 - San Francisco)
• 2011 - Introducing Puppi
Puppi: Puppet strings to the shell (PuppetCamp Europe 2011 - Amsterdam)
• 2012 - Example42 Next Gen modules
Developing IT Infrastructures with Puppet (CodeMotion 2012 - Rome)

3. WE ALL LOVE
AND USE PUPPET FOR
• Systems Configuration
• (Automatic) Monitoring based on specific tools
• Facts based Inventory
• Manage, at times, Applications deployments
• Infrastructure Orchestration (coupled with MCollective)

4. WE LIKE
TO EXTEND PUPPET TO
• Abstract Automatic Monitoring (whatever the tool)
• Automatic Firewalling
• Standardize Applications deployments
• Enrich Systems Inventory
• Shell Extension (“Puppet Knowledge to the CLI”)
• Provide a coherent and integrated modules ecosystem

5. PUPPET MODULES MANTRAS
• Data Separation
• Configuration data is defined outside the module (or Puppet manifests)
• Module’s behavior is managed via APIs
• Reusability
• ReUse the same module in different shops
• Customize its behavior without changing its code
• Do not force how configurations are provided
• Standardization
• Follow PuppetLabs layout guidelines (puppet-lint)
• Have a coherent, predictable and intuitive interface
• Provide contextual documentation (puppet-doc)
• Interoperability
• Limit dependencies. Allow modules’ cherry picking
• Be self contained, do not interfere with other modules’ resources
• Cross Operating System support
• Provide sensible defaults for different OSes
• Allow easy implementation of support of new OSes

6. EXAMPLE42 NEXT GEN
• Coherent and Standardized structure
• Best Practices module design (with some tweaks...)
• Easily extendable Cross OS support
• Complete API exposure via parameters
• Extreme Customizations options
• Alternative Data Separation options
• Complete Decommissioning features
• Optional Automatic Monitoring Abstraction
• Optional Automatic Firewalling
• Optional Puppi support to enhance the CLI experience
• Exhaustive PuppetDoc documentation
• Integrated Rspec-Puppet tests
• Code Puppet-Lint compliant
• Quick module scaffolding based on different templates
... not exactly easy to read....

7. BASIC USAGE
• One Module. One Application. One main class.
• Install openssh with default settings:
class { 'openssh': }
• Equivalent to:
include openssh
• Default behavior:
• Install package
• Run and enable service
• Do not alter configurations

8. DATA INPUT ALTERNATIVES
• Set (top scope/ENC) variables and include classes:
$::openssh_template = 'site/openssh/openssh.conf.erb'
include openssh
• Use Hiera:
hiera(‘openssh_template’)
include openssh
• Use parametrized classes:
class { 'openssh':
template => 'site/openssh/openssh.conf.erb',
}
• Happily mix different patterns:
$::monitor = true
$::monitor_tool = [ 'nagios' , 'munin' , 'puppi' ]
class { 'openssh':
template => 'site/openssh/openssh.conf.erb',
}

9. DECOMMISSIONING
• Disable openssh service:
class { 'openssh':
disable => true
}
• Disable openssh service only at boot time:
class { 'openssh':
disableboot => true
}
• Remove openssh (package and files):
class { 'openssh':
absent => true
}
• Monitoring and firewalling resources removal is automatically
managed

10. MANAGE BEHAVIOR
• Enable Auditing:
class { 'openssh':
audit_only => true, # Default: false
}
(No changes to configuration files are made and what would be done is audited)
• Disable service autorestart:
class { 'openssh':
service_autorestart => false, # Default: true
}
(No automatic service restart when a configuration file / dir changes)
• Manage software version:
class { 'foo':
version => ‘1.2.0’, # Default: unset
}
Specify the package version you want to be installed.
Set => ‘latest’ to force installation of latest version

11. CUSTOMIZE: CONFIGURATION FILE
• Provide configuration as a static file ...
class { 'openssh':
source => ‘puppet:///modules/site/ssh/sshd.conf’,
}
• an array of files looked up on a first match logic ...
class { 'openssh':
source => ["puppet:///modules/site/ssh/sshd.conf-${fqdn}",
"puppet:///modules/site/ssh/openssh.conf"],
}
• As an erb template:
class { 'openssh':
template => ‘site/ssh/sshd.conf.erb’,
}
• Config File Path is defined in params.pp (can be overriden):
config_file = >’/etc/ssh/sshd_config’,

12. CUSTOM OPTIONS
• With templates you can provide an hash of custom options:
class { 'openssh':
template => ‘site/ssh/sshd.conf.erb’,
options => {
'LogLevel' => 'INFO',
'UsePAM' => 'yes',
},
}
• Alternative ways to use the options hash in an erb template:
• Direct but not safe (you must always provide all the used options)
UsePAM <%= options['UsePAM'] %>
• Failsafe with defaults (verbose but safe)
<% if scope.lookupvar("openssh::options['UsePAM']") then -%>
UsePAM <%= options['UsePAM'] %>
<% else -%>
UsePAM no
<% end -%>
• Show what you have (useful for config files has defaults for every option)
<% scope.lookupvar("openssh::options").sort_by {|key, value| key}.each do |key,
value| -%>
<%= key %> <%= value %>
<% end -%>

13. CUSTOMIZE: CONFIGURATION DIR
• You can manage the whole configuration directory:
class { 'openssh':
source_dir => ‘puppet:///modules/site/ssh/sshd/’,
}
This copies all the files in lab42/files/ssh/sshd/* to local config_dir
• Youcan purge any existing file on the destination config_dir
which are not present on the source_dir path:
class { 'openssh':
source_dir => ‘puppet:///modules/site/ssh/sshd/’,
source_dir_purge => true, # default is false
}
WARNING: Use with care
• Config Dir Path is defined in params.pp (can be overriden):
config_dir = >’/etc/ssh’,

14. CUSTOMIZE: CUSTOM CLASS
• Provide added resources in a custom class:
class { 'openssh':
my_class => ‘site/my_openssh’,
}
This autoloads: site/manifests/my_openssh.pp
• Custom class can have whatever you may need to add:
class site::my_openssh {
file { "motd":
path => "/etc/motd",
content => template("site/openssh/motd.erb"),
}
}
You hardly need to inherit openssh: there are parameters for everything
Do not call your class site::openssh, bad things may happen.

15. CUSTOMIZE: PATHS AND NAMES
• An example: Use the puppet module to manage pe-puppet!
class { 'puppet':
template => 'lab42/pe-puppet/puppet.conf.erb',
package => 'pe-puppet',
service => 'pe-puppet',
service_status => true,
config_file => '/etc/puppetlabs/puppet/puppet.conf',
config_file_owner => 'root',
config_file_group => 'root',
config_file_init => '/etc/sysconfig/pe-puppet',
process => ‘ruby’,
process_args => ‘puppet’,
process_user => ‘root’,
config_dir => '/etc/puppetlabs/puppet/',
pid_file => '/var/run/pe-puppet/agent.pid',
log_file => '/var/log/pe-puppet/puppet.log',
log_dir => '/var/log/pe-puppet',
}

16. EXTEND: MONITOR
• Manage automatic monitoring:
class { 'openssh':
monitor => true,
monitor_tool => [ ‘nagios’,‘puppi’,‘monit’ ],
monitor_target => $::ip_addess # Default
}
• Monitoring is based on parameters defined in params.pp:
port => ‘22’,
protocol => ‘tcp’,
service => ‘ssh[d]’, # According to OS
process => ‘sshd’,
process_args => ‘‘,
process_user => ‘root‘,
pid_file => ‘/var/run/sshd.pid’,
• Abstraction is managed in the Example42 monitor module
Here “connectors” for different monitoring tools are defined and can be added (also using 3rd
party modules).

17. EXTEND: FIREWALL
• Manage automatic firewalling (host based):
class { 'openssh':
firewall => true,
firewall_tool => ‘iptables’,
firewall_src => '10.0.0.0/8',
firewall_dst => $::ipaddress_eth1, # Default is $::ipaddress
}
• Firewallig is based on these parameters defined in params.pp:
port => ‘22’,
protocol => ‘tcp’,
• Abstraction is managed in the Example42 firewall module
Currently only the “iptables” firewall_tool is defined, it uses Example42 iptables module to
manage local iptables rules

18. EXTEND: PUPPI
• Manage Puppi integration:
class { 'openssh':
puppi => true, # Default: false
puppi_helper => ‘standard’ # Default
}
• The Puppi module is a prerequisite for all Example42 modules
Is required because it provides common libs, widely used in the modules
BUT the actual puppi integration is optional (and disabled by default)
• Puppi integration allows CLI enrichment commands like:
puppi info openssh
puppi log openssh
puppi check openssh
Note: puppi support for info/log commands for NextGen modules is under development
• Puppi helpers allow you to customize puppi behavior

19. PARAMS_LOOKUP EVERYWHERE
• Each parameter on NextGen class is passed via params_lookup
class openssh (
[...] # openssh module specific parameters ...
$my_class = params_lookup( 'my_class' ),
$source = params_lookup( 'source' ),
$source_dir = params_lookup( 'source_dir' ),
$source_dir_purge = params_lookup( 'source_dir_purge' ),
$template = params_lookup( 'template' ),
$service_autorestart = params_lookup( 'service_autorestart' , 'global' ),
$options = params_lookup( 'options' ),
$version = params_lookup( 'version' ),
$absent = params_lookup( 'absent' ),
$disable = params_lookup( 'disable' ),
$disableboot = params_lookup( 'disableboot' ),
$monitor = params_lookup( 'monitor' , 'global' ),
$monitor_tool = params_lookup( 'monitor_tool' , 'global' ),
$monitor_target = params_lookup( 'monitor_target' , 'global' ),
[...] # Other common parameters
) inherits openssh::params {
[...]
}
• Different kind of params that:
• Are module specific (no one defined in this openssh module)
• Allow customizations (my_class, source, template ...)
• Affect module’s behavior (absent, disable, service_autorestart, audit_only ...)
• Manage extensions (monitor, monitor_tool, firewall, puppi ...)
• Define application data (port, config_file, process, package ... )

20. PARAMS.PP
• Each module has a params class where defaults are set for different OS
class openssh::params {
### Application related parameters
$package = $::operatingsystem ? {
default => 'openssh-server',
}
$service = $::operatingsystem ? {
/(?i:Debian|Ubuntu|Mint)/ => 'ssh',
default => 'sshd',
}
$process = $::operatingsystem ? {
default => 'sshd',
}
[...]
$port = '22'
$protocol = 'tcp'
# General Settings
$my_class = ''
$source = ''
$source_dir = ''
$source_dir_purge = ''
[...]
### General module variables that can have a site or per module default
$monitor = false
$monitor_tool = ''
$monitor_target = $::ipaddress
$firewall = false
$firewall_tool = ''
$firewall_src = '0.0.0.0/0'
[...]
}

21. PARAMS_LOOKUP ORDER
• params_lookup is a function provided by the puppi module
• It allows data to be defined in different ways:
• Via Hiera, if available
• As Top Scope variable (as provided by External Node Classifiers)
• Via defaults set in the module’s params class
• The “global” argument is used to define site_wide behaviour
• Example:
class { ‘openssh’:
monitor => true
} # If there’s a direct param that’s the value, otherwise:
# If hiera is available:
hiera(“monitor”) # If global lookup is set
hiera(“openssh_monitor”) # A module specific value overrides the global one
# If variable is still not evaluated:
$::monitor # If global lookup is set
$::openssh_monitor # If present, overrides $::monitor
$openssh::params::monitor # Module’s predefined value is used as default

22. DOWNLOAD
• Example42 Puppet Modules Site: http://www.example42.com
• GitHub repositories: http://github.com/example42
• Download:
git clone -r http://github.com/example42/puppet-modules-nextgen
• Note on GitHub repos:
• puppet-modules-nextgen contains only NextGen modules
• puppet-modules contains both NextGen and older modules

23. ONE MORE THING...
• How to make a NextGen module
git clone -r http://github.com/example42/puppet-modules-nextgen
cd puppet-modules-nextgen
Example42-tools/module_clone.sh
This script creates a skeleton for a new module based on different Example42 foo module
templates. Run it from the directory that contains the foo module (moduledir).
By default it uses the "foo" module as template.
Specify -t <source_module> to use a different template.
Example:
Example42-tools/module_clone.sh -t foo_webapp
Source module template is foo
Enter the name of the new module based on foo: mynewmodule
E d i t my n e w m o d u l e / m a n i fe s t s / p a r a m s . p p t o m a n a g e s u p p o r t fo r d i f fe r e n t
OSes
•A new module (with the features seen so far) based on the
foo standard template is done. Add features and application /
OS specific resources to enrich it