Like most IT companies, also we at Raiffeisenverband Südtirol noticed, that because of recent trends, the demand of new servers has grown exponentially in the last years. Simultaneously, the number of human resources, responsible for those servers, remained more or less stable. In order to still be able to handle the workload, we needed to automatize and centralize server management as much as possible. That’s where configuration management (CM) kicks in. CM is a very broad term and there are plenty solutions on the marked, Open Source and Enterprise, but the general approach is to make configurations in a central place, and then roll them out to the infrastructure, i.e., to one or more servers. We evaluated a few solutions, and finally decided that wedon’t want to manage all our servers with a single big fat monster, which nobody of us is able to debug if it goes crazy. But reinventing the wheel was not an option, so we decided to make use of a hand full Open Source Software components and integrated them to a CM solution, customized to our needs. The main components are Puppet, Git, Foreman and Pulp.

1. Config Management with
Puppet, GIT & some Ruby magic
Stefan Peer – System Engineer
11.11.2016

2.  Head organization of 369 cooperatives
 with more than 124.000 single members
 Service provider and consulting
 IT, HR, financial, legal, education and much more
 310 employees in total
 40% in IT
 Raiffeisen Informationssystem (RIS)
 IT service provider of the Raiffeisen Group
 Datacenters in Bolzano and Milano
2
Raiffeisenverband Südtirol

3.  Applications running on different platforms
 z/OS (Mainframe), Linux, Solaris, Windows
 Heavily rely on virtualization and automation
 VMware, Solaris container
3
IT Systems in RIS
5 5 6 6 7 7 7 8 8 8 8
0
100
200
300
400
500
600
700
800
900
1000
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
# VMs
# Administrators

4. Configuration Management is the process of
standardizing resource configurations and enforcing
their state across IT infrastructure in an automated yet
agile manner.
(Puppetlabs)
4
Definition

5.  Growth
 same effort to make a change on 1 or 1000 servers
 Central Governance
 in a heterogeneous environment with various OS
 Traceability / Reporting
 obliged by certifications such as PCI/DSS or ISAE3402
 Rollback
 revert changes
 Durability
 keep config-state consistent
 Consistent Environments
 hand over changes: Test => QA => Production
5
Why Configuration Management?

6. 6
Admin‘s daily life … before CM – part 1
Hey Linux!
We need to change the IP Address
of our secondary DNS server!
Okay, don't worry. Gimme a week.
Don’t have
time for that!
Hmm, I could write a
script that SSHes into
all our servers and
applies the change!
But what about
this other
Debian server?

7. 7
Admin‘s daily life … before CM – part 2
Hey Solaris!
We need to change the IP Address
of our secondary DNS server!
Okay, lot’s of manual work, but we
will have it done by next week!
Hmm, good
task for our
intern 
Damn!
Project delayed
for another week!

8. 8
Admin‘s daily life … with CM
Hey Linux!
We need to change the IP Address
of our secondary DNS server!
Ok, hang on, I’ll commit the change into CM.
Done, change will be rolled out within half an hour.
Btw. to Solaris servers as well!
Thanks man! Good work!
Where could
I go skiing
tomorrow?

9. 9
Let the puppets dance!
Puppet Master
1. facts
ex.
I am Frida, a
RHEL 6.8
with 2 cores
Foreman
4. reference config
ex. Apache must be running,
listening on Port 443
2. ask ENC
ex. who is Frida?
3. classes and params
ex. Apache server located
in Bolzano
each server,
every 30 minutes

10. 10
Let the puppets dance!
Puppet Master
6. report
ex.
service Apache
failed to start
Foreman
7. forward report
ex. service Apache failed
to start on Frida
5. apply reference config
ex. service httpd start
ex. for Solaris it would be:
svcadm enable /network/http:apache22

11.  Assign Puppet Classes to hosts (ENC)
 ex. Icinga Master host
 What are your servers doing?
 What has changed on server X?
11
Foreman

12. 12
What can I do with Puppet?
 Manage files
file {'/etc/httpd/conf/httpd.conf':
ensure => present,
content => template('${module_name}/httpd.conf.erb'),
owner => 'root',
group => 'root',
mode => '0644',
}
 Manage services
service {'httpd':
ensure => running,
enable => true,
}

13. 13
What can I do with Puppet?
 Install or uninstall software
package {'httpd':
ensure => installed,
}
 Execute commands
 Create Cron jobs
 Manage certificates and Java Keystores
 and much, much more …
package {'tcpdump':
ensure => absent,
}

14. 14
Puppet manifest
class ris_ftp::server (
$local_root_dir,
){
package { 'vsftpd':
ensure => installed
}
-> file { '/etc/vsftpd/vsftpd.conf':
content => template("${module_name}/vsftpd.conf.erb"),
notify => Service['vsftpd'],
}
service { 'vsftpd':
ensure => running,
enable => true,
}
}

15. 15
GIT – the place where all the Puppet code is stored

16.  One special GIT repo that connects everything together
 we call it „control-repo“
 GIT branch per environment
 New environment needed? Simply fork a branch!
16
Dynamic environment creation with r10k
Puppetfile
ris_dns => Commit 12
ris_ssh => Commit 3
ris_icinga => Commit 45
Puppetfile
ris_dns => Commit 11
ris_ssh => Commit 2
ris_icinga => Commit 40
Puppetfile
ris_dns => Commit 11
ris_icinga => Commit 36
Merge changes Merge changes
TEST QA PRODUCTION

17. 17
Puppetfile in real – and that‘s just a part of it

18.  History of our control-repo
 Including current state of each branch, .i.e., environment
18
control-repo in real

19.  Nearly impossible to manage control-repo + Puppetfile by hand
 That‘s why we wrote a Ruby toolset that helps us managing it
 we call it ris-puppet
 Examples:
 ris-puppet module validate
 ris-puppet module deploy --env=test
 ris-puppet environment create --env=stefan --from=production
 ris-puppet foreman import
 Integrated also in GIT server via hooks
 ex. reject commit if there are syntax errors
19
Now, where‘s the Ruby magic?

20. 20
Questions ?