Inspired by the talk "Using Rundeck and Chef to build DevOps Toolchains" given by Anthony Shortland during ChefConf 2012 (http://dev2ops.org/2012/05/using-rundeck-and-chef-to-build-devops-toolchains-at-chefcon/) we set out to include Rundeck as part of a loosely coupled toolchain when building the new datacenter at Bol.com. This talk explored how we took the ideas from Shortland's talk and implemented them in the context of Jenkins, Gerrit, Puppet and MCollective.
Tamil Call Girls Bhayandar WhatsApp +91-9930687706, Best Service
Building a loosely coupled toolchain with Rundeck and Puppet
1. Outline Goal A Problem to Solve Tools Practical Examples
Building a loosely coupled toolchain with
Rundeck and Puppet
Steven Meunier
21 June 2014
2. Outline Goal A Problem to Solve Tools Practical Examples
1 Goal
What are we trying to achieve?
Influences
Infrastructure
2 A Problem to Solve
3 Tools
What do we need
Something to trigger a change
Somewhere to store the data for the change
Something to perform the change
Helping the tools help us
4 Practical Examples
Generic Deployment Flow
Updating Hiera Data
Rundeck
3. Outline Goal A Problem to Solve Tools Practical Examples
Goal
To have a loosely coupled deployment toolchain that leverages
our desire to manage the servers based on state
Implies decisions have already been made regarding kind of
tooling and server management:
Tools that perform a particular function and do it well
Tools can be swapped out if needed
Concept of state can be extended to deployments if viewed
as a process of changing the server state
4. Outline Goal A Problem to Solve Tools Practical Examples
Influences
Deployment models from glu
Infrastructure as data from ansible
Puppet server state
http://dev2ops.org/2012/05/using-rundeck-and-chef-to-
build-devops-toolchains-at-chefcon/
5. Outline Goal A Problem to Solve Tools Practical Examples
Normal Puppet Infrastructure
Puppet Infrastructure
6. Outline Goal A Problem to Solve Tools Practical Examples
Deployment Infrastructure Built On Top Of Puppet
Deployment Infrastructure
7. Outline Goal A Problem to Solve Tools Practical Examples
Ready . . . Set . . .
By tableatny (BXP135671) [CC-BY-2.0], Source:
http://upload.wikimedia.org/wikipedia/commons/1/13/Athlete_at_starting_block.jpg
8. Outline Goal A Problem to Solve Tools Practical Examples
Go!
By David A. Brandenburg, U.S. Navy (http://www.navy.mil/; VIRIN: 100517-N-7643B-048) [Public domain], Source:
http://upload.wikimedia.org/wikipedia/commons/2/2a/Sack_race.jpg
9. Outline Goal A Problem to Solve Tools Practical Examples
Know your Problem Domain
Deadlines
Restrictions imposed by Security Team
Team requirements
Release management (easy interface, keep track of
versions, support all our applications and release process)
Development (APIs for jenkins, low (preferably no) adoption
barrier for development)
Operations (operations maintain control, integrate with
puppet, mcollective)
10. Outline Goal A Problem to Solve Tools Practical Examples
What do we need?
Something to trigger a change
Somewhere to store the information for the change
Something to perform the change
11. Outline Goal A Problem to Solve Tools Practical Examples
Something to trigger a change
Rundeck
Rundeck is an automation console
Makes use of existing puppet infrastructure
Can automate whatever we want, not just deployments
Fine grained access controls
Easily extensible through plugins — even if you don’t know
Java
Deployments triggered manually or via Jenkins
12. Outline Goal A Problem to Solve Tools Practical Examples
Somewhere to store the data for the change
Hiera data
“Infrastructure as code” → “Infrastructure as data”
Hiera data for configuration using Puppet 3’s data binding
Change configuration without editing puppet manifests or
templates
property_files::parent_prefix: /srv/tomcat6/foo/conf
property_files::property_files:
- foo.properties
- bar.properties
properties_foo.properties:
database.name: fooDB
timeout: 100
13. Outline Goal A Problem to Solve Tools Practical Examples
Something to perform the change
Puppet & MCollective
Puppet manages all aspects of the server — nothing is
manual
Mcollective is an orchestration tool using activemq to
trigger agents across servers
MCollective manage non-state managed aspects or
temporary state changes
starting/stopping queues
database deployments
14. Outline Goal A Problem to Solve Tools Practical Examples
No matter what you think you’ve got...
Spaceballs, c 1987 MGM Studios, Source:
http://s3.amazonaws.com/mgm-assets/assets/Image/1266267325/41135/SPACEBAL-still-003-x.jpg
15. Outline Goal A Problem to Solve Tools Practical Examples
Sometimes your princess is a hairy guy in a dress
Spaceballs, c 1987 MGM Studios, Source:
http://img.pandawhale.com/65424-Spaceballs-Celebrity-stunt-dou-DZzt.jpeg
16. Outline Goal A Problem to Solve Tools Practical Examples
Tools need help to make them fit
Git: steep learning curve
Rundeck: needs extra plugins to fit your workflow
Puppet: duplicate declarations; ordering; limitations of
hiera
MCollective: can be unreliable if used as a source of node
truth; time needed to build the agents
Ruby: it’s critical if you’re using puppet + mcollective
17. Outline Goal A Problem to Solve Tools Practical Examples
Generic Deployment Flow
Create deployment window with no puppet interference
Build rpms from deployable unit (if applicable)
Disable monitoring
Compare and update configuration (if applicable)
Update version parameter in hiera data
Kick off puppet run
Restart service
Re-enable monitoring
18. Outline Goal A Problem to Solve Tools Practical Examples
Deployment Configuration Committer
Ruby script with the goal of reducing manual actions
related to configuration
Works with java properties files and context xml files
Builds a hash of properties in the release and compares it
against the current properties
Uploads any changes to Gerrit for review
19. Outline Goal A Problem to Solve Tools Practical Examples
An example tomcat application in Hiera
tomcat_context:
ROOT:
version: xx.yy.zz
tomcat_instance:
foo: {}
tomcat_context:
ROOT:
artifact: foo
artifact_provider: file
artifact_target: ROOT.war
artifact_url: http//repo.example.com/war
instance: foo
rundeck_job:
foo: {}
20. Outline Goal A Problem to Solve Tools Practical Examples
Puppet Rundeck Job Type
@@rundeck_job { "${project}/${group}/${name}_${hostname}":
ensure => $ensure,
jobname => $name,
project => $project,
group => $group,
definition => template($template),
}
21. Outline Goal A Problem to Solve Tools Practical Examples
Rundeck Conditional Local Command
#!/bin/sh
COMMAND_TRUE=$1
COMMAND_FALSE=$2
CONDITION="[ $3 ]"
EXIT_CODE=0
if [ "x$COMMAND_FALSE" == "x" ]
then
if eval $CONDITION
then
$COMMAND_TRUE
EXIT_CODE=$?
fi
else
if eval $CONDITION
then
$COMMAND_TRUE
EXIT_CODE=$?
else
$COMMAND_FALSE
EXIT_CODE=$?
fi
fi
exit $EXIT_CODE
22. Outline Goal A Problem to Solve Tools Practical Examples
Rundeck Conditional Local Command
23. Outline Goal A Problem to Solve Tools Practical Examples
Thank You