Uphill battle against large enterprise it environments and IT corporate culture. How those difficulties turned out opportunities and clever implementations. Interesting modules, integrations and workflow pieces.

1. Workflow story –
theory versus practice
in Large Enterprises
Marcin Piebiak
Solutions Architect
Linux Polska Sp. z o.o.
1

2. What is it?
●
Resources
A lot of ready to use resources:
• user
• group
• host
• cron
• exec
• file
• package
• service
• ...
package { 'ssh':
ensure => 'installed',
name
=> 'openssh-server',
}
service { 'ssh':
ensure => 'running',
name
=> 'sshd',
require => Package['ssh'],
}
• Resources are building blocks.
• They can be combined to make larger components.
• Together they can model the expected state of your system.
2

3. What is it?
Resources
● Declarative language
●
package { 'ssh':
ensure => 'installed',
name
=> 'openssh-server',
}
service { 'ssh':
ensure => 'running',
name
=> 'sshd',
}
Rather than listing a series of steps to carry out
we can describe the desired final state only.
3

4. What is it?
Resources
● Declarative language
● Abstraction
●
package { 'ssh':
ensure => 'installed',
name
=> 'openssh-server',
}
root@debian ~]# apt-get install openssh-server
root@redhat ~]# yum install openssh-server
Resources in Puppet are
abstracted from
underlying providers.
4

5. What is it?
Resources
● Declarative language
● Abstraction
● Facts
●
Puppet uses facter to gather
information about the host system.
root@redhat ~]# facter
architecture => x86_64
domain => linuxpolska.pl
facterversion => 1.5.2
fqdn => redhat.linuxpolska.pl
hardwaremodel => x86_64
hostname => redhat
interfaces => eth0
ipaddress => 172.16.10.1
kernel => Linux
...
5
Custom Facts
Facter.add('role') do
setcode do
'cat /etc/role'
end
end

6. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
●
Puppet uses Hiera as its single source
of truth data abstraction layer.
$pkg_name = hiera('pkg_name')
package { 'apache':
ensure => 'installed',
name
=> $pkg_name,
}
Hiera uses Facter facts to determine
a hierarchy.
6
--:backends:
- yaml
:yaml:
:datadir:/etc/hiera
:hierarchy:
- %{fqdn}
- %{osfamily}
- %{environment}
- common

7. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
●
class ssh (
$pkg_name = 'openssh-server',
$srv_name = 'sshd',
) {
package { 'ssh':
ensure => 'installed',
name
=> $pkg_name,
}
}
7
service {
ensure
enable
name
require
}
'ssh':
=> 'running',
=> 'true',
=> $srv_name,
=> Package['ssh'],
Classes define a
collection of
resources that
are managed
together as a
single unit.

8. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
●
Classes are abstracted by modules:
ssh
├──
│
│
├──
│
└──
manifests
├── init.pp
└── server.pp
files
└── ssh_config
templates
└── sshd_config.erb
Modules are directories that contain your
configuration. They are designed to encapsulate all
of the components related to a given configuration in
a single folder hierarchy.
8

9. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
●
If a more complex deployment is
needed, reusing existing classes
saves effort and reduces error.
9

10. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
●
You own abstraction layers...
Profiles:
class profiles::application {
include tomcat
include mysql
include myapp
}
class profiles::base {
include ssh
include ntp
include users
}
Roles:
class role::webapp {
include profiles::base
include profiles::customapp
include profiles::test_tools
}
10

11. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
● Supports many OS
●
Supported OS
11

12. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
● Supports many OS
●
Big environment – no problem
Changes?
base/ntp.yaml
--ntp::local_servers:
- 192.168.0.45
+ - 192.168.0.1
12

13. What is it?
Resources
● Declarative language
● Abstraction
● Facts
● Data separation
● Reusable code
● Supports many OS
● Provisioning
● Orchestration
● Puppet Forge
● Live Management
● Environments
● Reporting
● Dry-run mode
●
And more...
VM/Cloud Provisioning
Live Management
Orchestration
MCollective
Environments
Dry-run mode
13
Reporting
Custom:
- types & providers
- facts
- functions

14. What is it?
Resources
● Declarative language
Thanks to these all superior features, ● Abstraction
Puppet is:
● Facts
- fast in deployment
● Data separation
- easy to use
- universal for a lot of operating systems ● Reusable code
● Supports many OS
- with unlimited possibilities
● Provisioning
- easy to expand
- flexible
● Orchestration
● Puppet Forge
● Live Management
● Environments
● Reporting
● Dry-run mode
●
14

15. In large enterprises like banks, telco, insurance, etc. those features
are not the most relevant.
Implementing Puppet in enterprises we must consider another
priority map, another mindset.
VS.
We must answer not trivial questions dealing
with the core IT Departments way of work.
15

16. So... everyone has
access to everything?!
VS.
16

17. So... can I destroy
the whole infrastructure
from one place?
VS.
17

18. Solution
Puppet master installation
on hardened system with
limited direct access.
tcp:8140
RBAC
tcp:443
tcp:22
ssh keys
For maintenance.
18

19. Solution
Release manager
Pull request
developer
developer
Fetch
Git as a communication
layer between developers
and puppet master.
19
developer

20. So... everyone now must
use puppet?!
VS.
20

21. Developers
Solution
Systems
administrators
Databases
administrators
Security
department
On beginning each department
can have own environments.
21

22. Each department can have
many environments and its
own idea how to organize
work with puppet.
Solution
Security
department
Integrator
Tests
integrator
dev
test
prod
Release
manager
22

23. Each department can have
many environments and its
own idea how to organize
work with puppet.
Solution
Security
department
Integrator
Tests
integrator
Release
v0.1
v0.2
dev
test
prod
commits
23
Release
manager

24. Developers
Solution
Systems
administrators
Databases
administrators
integrator
Security
department
After time some departments will start
working together in one environment.
24

25. Developers
Solution
Systems
administrators
Databases
administrators
integrator
Security
department
integrator
25

26. Developers
Solution
Systems
administrators
Databases
administrators
integrator
Security
department
integrator
At the end all departments will use
one environment.
26

27. How can I find out
who made what change
and who approved
this modification?
VS.
27

28. Using git we have:
● date
● author
● description
Solution
commit 220938c5a2e51ecd4166eb7d75d14974cbcff897
Author: Marcin Piebiak <mpp@linuxpolska.pl>
Date:
Fri Jul 5 11:27:43 2013 +0200
Description....
Person who approved
modifications.
Release
v0.1
dev
test
prod
28
v0.2
We can use git as a
place for history of the
infrastructure.
● git status
● git log
● git diff
commits
● git blame
Git history is cryptographically secured.

29. If I have a lot of
environments how can
I use them?
VS.
29

30. Solution
We can specify a set of
environments for each host to
use.
Database testing system, uses
three testing environments from
different departments.
tcp:8140
dev
test
prod
30

31. Great!
But... using command
line I can connect to
different environments!
VS.
31

32. Solution
We use imp module, to control
puppet agents behavior and
their access to environments.
puppet agent -t --environment
tcp:8140
32

33. Solution
puppet.conf
[main]
modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
manifest
= /etc/puppetlabs/manifests/site.pp
[env_sec_prod]
modulepath = /var/lib/git/env_sec_prod/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
[env_sec_test]
modulepath = /var/lib/git/env_sec_test/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
/etc/puppetlabs/puppet/modules – place for modules from forge and well tested modules and module imp
/var/lib/git/$environment/modules – git repository for environment
/etc/puppetlabs/manifests/site.pp - common manifest for all environments
imp.yaml
site.pp
include imp
hiera_include('include')
33
--imp::environments:
env_sec_prod:
order: 'deny,allow'
deny: 'all'
allow:
- 'host1.linuxpolska.lab'
- 'www.*.linuxpolska.lab'
commiters:
- 'john.smith'
priority: 1

34. How can I audit
changes made in
the infrastructure?
VS.
34

35. Puppet reports
Solution
When nodes fetch their
configurations from the puppet
master, they send back
inventory data and a report of
their run.
puppet reports
Log collector
and analyzer.
35

36. How can puppet help
me with audit? How can
I recreate life cycle
of each host?
VS.
36

37. Solution
puppet reports
● puppet agents catalog
● hosts facts
● git diffs after commit
● hiera configuration for
each host
● filebuckets
● ...
●
Log collector
and analyzer.
37

38. Solution
file {'/etc/important':
ensure => 'file',
group => 'apache',
mode
=> '0660',
}
app
db
file {'/etc/important':
ensure => 'file',
user
=> 'root',
group => 'root',
mode
=> '0600',
}
system
security
If we have many environment there is always
risk of overwriting someone's changes.
Log collector
and analyzer.
38

39. How will the
modification in puppet
manifests
affect the whole
infrastructure?
VS.
39

40. Solution
Using log collector we can analyze the infrastructure modifications
before they get to production environment.
Report from puppet normal run.
Report from puppet dry-run.
v0.1
v0.2
dev
test
prod
commits
Log collector
and analyzer.
40

41. All changes are made
automatically? First I would
like to see what is
going to be changed.
VS.
41

42. How can we rollback
changes?
VS.
42

43. After we install
puppet will we know
everything about
the infrastructure?
VS.
43

44. THE END
Marcin Piebiak
Solutions Architect
Linux Polska Sp. z o.o.
44