1. RMLL 2011 @Strasbourg, France
A tale of disaster recovery
CFEngine everyday, practices and tools
Nicolas Charles <nch@normation.com>
Jonathan Clarke <jcl@normation.com>

2. About the speakers
Nicolas Charles Jonathan Clarke
CFEngine contributor CFEngine contributor
CFEngine ”Community Contributor to various
Champion” (C3) LDAP FLOSS projects
Scala Developer Sysadmin
But we get on pretty well!
(mostly...)

3. Agenda
1) Configuration Management 101
2) A tale of disaster recovery
3) Our choice of tool
4) About CFEngine 3

4. A bit about
Configuration Management...

5. Configuration management
 What is it?
 Configuration Management is a field of
management that focuses on establishing and
maintaining consistency of a system (..)
throughout its life
 Software configuration management is the task
of tracking and controlling changes in the
software
Sources:
http://en.wikipedia.org/wiki/Configuration_management
http://en.wikipedia.org/wiki/Software_configuration_management

6. A server crashed.
Install a new one, people
can't work without it!
OK, it'll be done in
about two days...
Why configuration management?
There's a new critical security patch
we must deploy on all our servers!
Get it out quickly!
Right, I'll put the whole
team on it.

7. Reproducibility Industrialization
Automation
Why configuration management?

8. How do we setup
service X?
Ask Jim, he's
the expert on that.
But he left the company...
Why configuration management?
Huh, this server has been logging
errors for a few weeks.
Oh? I think Michael changed
something on it recently...
He'll tell you what it was.
Damn, he's on vacation!

9. Documentation History
Building-up
knowledge
Why configuration management?

10. An intruder just stole our data
using a vulnerability in a
module we don't need...
I thought the project specification
ensured that we disabled that?
Er, it did, but we enabled it to
solve a problem and forgot to
disable it afterwards... sorry...
Why configuration management?

11. Why configuration management?
Vigilance
Automatic repairs Alerts

12. I don't understand how this
server is setup. It doesn't match
our best-practices.
Oh, that's a legacy server...
Why configuration management?
Give me details on our
current security policy.
Well, it's a collection of little
things, here and there...
Ah... Well, OK.
Tell me: is it fully applied
on all our critical servers?
Er...

13. Why configuration management?
Rationalization
Normalization Control

14. Reproducibility Industrialization Documentation History
Automation Building-up
knowledge
Configuration management
Vigilance Rationalization
Automatic repairs Alerts Normalization Control

15. Disaster Recovery
An ill-fated tale
from the recent past
(CASE STUDY)

16. Before the disaster...
 Our company's IT infrastructure
 Small company: small requirements
 Web site, email
 Git repository, Redmine...
 Small company: small budget
 All on one hosted server

17. Asking for trouble?
 Just one hosted server! Critical services!
 No, a ”safe” configuration:
 Redundant hardware, 3 disk RAID-5 array
 All services automatically installed and setup
using Configuration Management
 Backups: daily (several off-site locations)
 Several VMs to separate services

18. A critical failure
 2 hard drives fail simultaneously
→ RAID-5 array is down
→ Almost all services fail immediately
→ ”The end of the world as we know it”
→ Need to rebuild everything NOW

19. Recovering
 Step 1: Panic!
 Step 2: Get a new server
 Step 3: Reinstall base OS + virtualization
 Step 4: Restore VM configuration whoops
 Step 4: Re-create the VMs manually
 Step 5: Reinstall each OS in each VM...

20. Recovering
 Step 6: Installation Configuration Management
 Step 7: Sit back and watch all the services
coming back online as if by magic!
 Step 8: Huh, where's my data?
 Step 9: Manually restore backups
 Step 10: Make a list of missing data...

21. Lessons learned
1) Hard disks fail reliably
2) Restoring virtualization setups:
● Backing up the config files would have helped
● Need CM tools to describe the desired state!
(Cfengine Nova does this)
3) Configuration Management should tie in to our
backup system
4) Backups were lacking some files: always test!

22. Wishlist and discussion
 Integrating Configuration Management tools
and backup systems is a crucial step for CM
to be efficient for disaster recovery
 What do others do?
 Provisioning VMs and their resources (disks,
network) should be automated too
 Cloud providers are one solution
 What about ”plain” virtualization?

23. Configuration Management
Tools
What we chose, and why

24. Our choice
 Back in mid 2009
 Needed a configuration management tool
 Criteria:
 Open source
 Multi-platform agent (including Windows)
 Resilient
 Non-disruptive

25. Our choice: candidates
CFEngine 3 Puppet Chef

26. Our choice: candidates
CFEngine 3
More on this
choice later...

27. A bit about CFEngine 3...
Sources: across the Internet

28.
CFEngine: History
Source:
http://verticalsysadmin.com/blog/uncategorized/relative-origins-o
f-cfengine-chef-and-puppet

29. CFEngine 3: Intro
 Configuration management software
 Written in C
 Two versions :
 Community (GPL v3)
 Nova (closed source)
 Community + extra features
 Some features released in Community
 Backed by CFEngine AS – Norway based
company founded in 2009

30. CFEngine 3: Features
Multi platform
Multi-agent technology Adapted to
Lightweight, non-intrusive heterogeneous
environments
Autonomous
Fault-tolerant Highly scalable
Progressive
roll-out
Large user base and community

31. CFEngine 3: Components
 Cf-agent
 Runs on all managed hosts
 Applies configuration – this is the heart
 Can connect to cf-serverd to get policies / files
 Cf-serverd
 Distributes policies and files
 Must be run on policy server(s)
 Usually run on all hosts to enable remote runs
 Cf-monitord
 Collects statistics on all nodes

32. Memory usage
 Daemon consumption on managed hosts

33. CFEngine 3: Usage examples
 Large companies
 Critical systems: Joint Australia Tsunami
Warning Centre
 Personal computers
 Mobile devices: Nokia N900
 Underwater devices: army submarines
 Small and medium companies...
 Community

34. Feature: Multi-platform
 Define a configuration for all operating systems
 Windows, Linux
 Make it ”transparent” (forget about the
complexity)
 Existing standard library handling the
differences between each OS and distribution

35. CFEngine 3: Promises
 Configuration rules are called promises
 ”Promise” to be in the desired state
 Cfengine agent handles the steps to get there:
convergence
 Promise theory is based on research done in
the University of Oslo

36. Feature: File editing
 Only change what you need to
 You like your distribution's defaults?
 You have various different systems already
setup and just need to change something?
 Search for lines and replace/delete/add them
 Only change one field in a file
 /etc/passwd for example

37. Feature: Complex tasks
 Powerful class system to trigger promises
 Based on nodes itself
 Based on time
 Based on whatever you might imagine
 Complex workflow can be created

38. Configuration example
 Install the LAMP stack
bundle agent caller {
vars:
"pkg_list" slist => { "httpd", "php5", "mysql" };
packages:
"${pkg_list}"
package_method => generic,
package_method => "addupdate";
}

39. Thank you !
RMLL 2011

40. CFEngine 3: Features
 According to Kuleven comparative study of
configuration management systems:
 Very mature
 Cross platform (*BSD, AIX, HP-UX, Linux, Mac
OS X, Solaris, Windows)
 Strongly distributed
 Based on state description and convergence
 Very high scalabily ( > 10000 nodes )
 Very small footprint
Source: http://distrinet.cs.kuleuven.be/software/sysconfigtools/overview