Mwrc2011 cookbook design patterns

Cookbook Design Patterns
Speaker:

Joshua Timberman Sr. Technical Evangelist
‣ joshua@opscode.com
‣ @jtimberman
‣ www.opscode.com
Copyright © 2010 Opscode, Inc - All Rights Reserved 1
Thursday, March 17, 2011

Welcome

Copyright © 2010 Opscode, Inc - All Rights Reserved
http://www.ﬂickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

cider:~/dev/cookbooks% git log | grep -ic "timberman"
950


I write cookbooks


Training, services and evangelism

Developers?
Systems Administrators?
Developers who do system
administration?
“Business” People?

http://www.ﬂickr.com/photos/timyates/2854357446/sizes/l/


Enough about me, who are you?


Lets talk about chef

http://www.ﬂickr.com/photos/tambako/4444066932/

Show of hands time! How many people....

...solo?


Who is using chef solo, Directly on your own or through a service like EY?

...server?


Who’s using their own Open Source chef server?

...Platform?


Who is using the Opscode Platform?

Chef enables infrastructure as code
Manage conﬁguration as idempotent Resources.
Put them together in Recipes.
Track it like Source Code.
Conﬁgure your servers.

package "haproxy" do
action :install
end

template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end

service "haproxy" do
action [:enable, :start]
end


This isn’t a talk about how Chef works or the nitty gritty, we assume that you have some
familiarity with Chef already. And there’s lots of resources for learning :).

wiki.opscode.com
help.opscode.com
joshua@opscode.com
@jtimberman

http://www.ﬂickr.com/photos/38299630@N05/3635356091/


If you want to know more, ﬁnd me

Chef provides an MVC
framework


Chef is an MVC framework for building infrastructure. How is that?

{
"kernel": {
"machine": "x86_64",
"name": "Darwin",
"os": "Darwin",
"version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53
PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386",
"release": "10.4.0"
},

Node attributes are the
"platform_version": "10.6.4",
"platform": "mac_os_x",
"platform_build": "10F569",

model
"domain": "local",
"os": "darwin",
"current_user": "jtimberman",
"ohai_time": 1278602661.60043,
"os_version": "10.4.0",
"uptime": "18 days 17 hours 49 minutes 18 seconds",
"ipaddress": "10.13.37.116",
"hostname": "cider",
"fqdn": "cider.local",
"uptime_seconds": 1619358
}

Models are data. Attributes are data. We’re going to process and mold the data to get to
where we want to be.

A configured node is the
view
Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/peterrosbjerg/3913766224/ 15

And where we want to be is a configured node. Running a Rails app, database, middleware,
whatever.

Recipes are the
controller

http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

They do all the processing of the data to build the view.

Recipes are Ruby.

Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.ﬂickr.com/photos/thisisbossi/3526698689/ 17

Since recipes are Ruby, that gives us a lot of power and ﬂexibility.

Cookbooks are
packages of recipes

http://www.ﬂickr.com/photos/riggenransom/4140166239

Recipes and supporting code, assets, etc.

Design patterns are
applicable to cookbooks


Since recipes and other things are code, and we’re really talking about infrastructure as code,
there’s good design patterns!

Cookbooks represent
best practices.


Best practices are
opinions.


Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.ﬂickr.com/photos/peyri/304354485
22

So lets talk about cookbooks


Of course I really mean Opscode’s cookbooks


Question: fork/clone/watch? Thanks!
Contributed? You’re awesome, thank you!
Tried to contribute and I told you to sign a CLA? Apache license requirement

http://www.opscode.com/blog/2009/08/11/why-we-
chose-the-apache-license/


Please sign a CLA, its for you as much for us


This is the important part: it doesn’t assign copyright to Opscode, you enable us to bundle
and redistribute your work, which means your code reaches a lot of people, yay!


But you can also have your code reach a lot of people through Opscode’s Chef Community
site: Its like RubyGems for Chef cookbooks. You don’t need to sign a CLA.

Its also easier to ﬁnd cookbooks than trawling through github.

Cookbooks package configuration

README
attributes/
definitions/
files/
libraries/
metadata.rb
providers/
recipes/
resources/
templates/

Cookbooks can have a bunch of stuff in them. Lets talk about this stuff.


Question: Who saw this blog post?

It talks about how writing your readme ﬁrst to get clarity about the code, plus it gives a nice
bit of documentation.

No really, write the ﬁne
manual.

http://www.ﬂickr.com/photos/bike/2263136517

Writing Recipes


Okay, you got writing the README out of the way, or maybe you didn’t. Now write the
recipes.

Contrived Example


A real world example


So I was at FOSDEM this year. Did anyone go?

Fosdem is a huge free conference in Europe, 5000+ people, 300 talks, 2 days.

GNU parallel

http://www.gnu.org/software/parallel/

It replaces xargs


One of the 300 talks was on GNU parallel.

I wrote a cookbook
during the talk.
http://www.ﬂickr.com/photos/jenorton/2229437427/

This isn’t because I’m
awesome.


http://www.ﬂickr.com/photos/ﬂikr/131433774/ 37

Cats with laser eyes are awesome.

Nor because the talk
was boring.


I wanted to write one
anyway.


GNU Parallel is like almost any GNU software.

wget ftp://ftp.gnu.org/gnu/thing/thing.tar.gz
tar -zxvf thing.tar.gz
cd thing
./conﬁgure && make && make install
Rejoice!


Do that in a recipe

remote_file "/tmp/parallel-20110205.tar.bz2" do
source "http://ftp.gnu.org/gnu/parallel/parallel-20110205.tar.bz2"
end

bash "build gnu parallel from source" do
cwd "/tmp"
code <<-EOF
tar -jxvf parallel-20110205.tar.bz2
(cd parallel-20110205 && ./configure)
(cd parallel-20110205 && make && make install)
EOF
end


So do that in a recipe.

That wasn’t hard.


Big deal, thats not hard. I can do that in about 42 seconds. And I did.

But thats a horrible
recipe.


Not because it installs from source. Unless..

'-. .-'
_______________'-._________.-'______________
'-. _ '-. .-' _ .-'
'-. (_) / (_) .-'
'-. / .-'
'-.____/ ____.-'
_ _ _ _ _ /
//////////
///////////
|||| .-----------._||||
|||| '-|___|___|-' ||||
'---------' ////
|||||||||||||///
///////
///// jnh

http://triple-double-u.com/ascii/?s=ascii-art&y=weather&q=ab/beard.txt
http://ascii-art.de/info/copyright/

Unless you have a unix sysadmin beard :) and package everything.

Really, why is it bad?


remote_file problems
Can’t download from a different location!
Can’t download a different version!
The file gets downloaded every time to verify the checksum.

Use customizable attributes

remote_file "/tmp/parallel-20110205.tar.bz2" do
source "http://ftp.gnu.org/gnu/parallel/parallel-20110205.tar.bz2"
end


This resource doesn’t have an easy way for someone to customize the location, or the
version, and the file has to be downloaded in order to verify the checksum every time.

Design Pattern 1:
Attributes


So we reach our ﬁrst design pattern, use attributes.

In cookbooks/gnu_parallel/attributes/default.rb:

default['gnu_parallel']['url'] = 'http://ftp.gnu.org/gnu/parallel'
default['gnu_parallel']['version'] = '20110205'
default['gnu_parallel']['checksum'] = 'sha256s dont fit on slides'


Wait, but I know what versions, and urls and everything in my infrastructure.

Yeah but if you want to share that, random folks on the internet don’t have the same
infrastructure.

/tmp isn’t a great location
Some systems clear it on reboot.
This causes the ﬁle to be downloaded again.
Use Chef::Config[:file_cache_path].


Downloading to /tmp isn’t a great solution either.

Design Pattern 2:
Exploit Chef internal
values


Second pattern, leverage the ability to access Chef’s Ruby objects in Recipes.

remote_ﬁle using attributes

version = node['gnu_parallel']['version']
cache_path = Chef::Config[:file_cache_path]

remote_file "#{cache_path}/parallel-#{version}.tar.bz2" do
source "#{node['gnu_parallel']['url']}/parallel-#{version}.tar.bz2"
checksum node['gnu_parallel']['checksum']
mode 0644
end


So now we have a remote ﬁle resource that uses some attributes, and internal Chef values.

bash script problems
bash "build gnu parallel from source" do
cwd "/tmp"
code <<-EOF
tar -jxvf parallel-20110205.tar.bz2
(cd parallel-20110205 && ./configure)
(cd parallel-20110205 && make && make install)
EOF
end

Least of all is compiling from source :-)


The other resource in the recipe has problems.

bash script solutions!
The version needs to be an attribute.
The default configure options may not be useful.
Moar attributes.

default['gnu_parallel']['configure_options'] = []


We should reuse the version attribute, which we saw earlier. What if we want to customize
how things are compiled, or the install prefix?

Define that as an attribute and set it to an empty array, user can modify the attribute in a role
or on the node.

Design Pattern 3: Sane
defaults easily changed


Empty conﬁgure options is sane because there aren’t any of those enabled when you do a
normal ./conﬁgure when compiling from source.

bash script using attributes

config_opts = node['gnu_parallel']['configure_options'].join(" ")

bash "build gnu parallel" do
cwd Chef::Config[:file_cache_path]
code <<-EOF
tar -jxvf parallel-#{version}.tar.bz2
(cd parallel-#{version} && ./configure #{config_opts})
(cd parallel-#{version} && make && make install)
EOF
end


So here we add a little more ruby, but gain a lot more ﬂexibility. We’re Rubyists, so we’re not
scared of Ruby.

'-. .-'
_______________'-._________.-'______________
'-. _ '-. .-' _ .-'
'-. (_) / (_) .-'
'-. / .-'
'-.____/ ____.-'
_ _ _ _ _ /
//////////
///////////
|||| .-----------._||||
|||| '-|___|___|-' ||||
'---------' ////
|||||||||||||///
///////
///// jnh

http://triple-double-u.com/ascii/?s=ascii-art&y=weather&q=ab/beard.txt
http://ascii-art.de/info/copyright/

The README tells his guy how to modify the attributes to customize how to conﬁgure and
where to install.

... Of course he’s going to make a package anyway.

Attribute conditional on platform in attributes ﬁle
case node['platform']
when "centos"
default['gnu_parallel']['install_method'] = 'package'
else
default['gnu_parallel']['install_method'] = 'source'
end

cookbooks/gnu_parallel/default.rb

include_recipe "gnu_parallel::#{node['gnu_parallel']['install_method']}"

cookbooks/gnu_parallel/package.rb

package "gnu-parallel"


Select which recipe to use based on platform, on centos we’ll install from package using the
package recipe

Design Pattern 12:
Platform speciﬁc
conditionals


Its good to utilize chef’s ability to look up the node’s platform and select behavior or set
attributes based on the platform.

INFO: remote_file[/var/cache/chef/parallel-20110205.tar.bz2]:
Creating /var/cache/chef/parallel-20110205.tar.bz2
INFO: Setting mode to 644 for remote_file[/var/cache/chef/
parallel-20110205.tar.bz2]
INFO: Ran bash[build gnu parallel] successfully


All that said, we can run chef on the node and get the source installed gnu-parallel, and
share this cookbook with other users who can use it how they wish. Yay!

All that in ~30 minutes


http://www.ﬂickr.com/photos/ﬂikr/131433774/ 61

Because I really am a Cat with Laser Eyes!

http://www.ﬂickr.com/photos/rutty/438775617 62

Here’s a giant rubber duck. Quack.

Recipes


Lets talk about Recipes a bit more. We saw a contrived example recipe, lets look at how to
best to utilize recipes in cookbooks. These are where we formulate our opinions.

Recipes

Separate by functionality
default
client
server
... etc


It is totally okay to have separate recipes split up by functionality of what they’re conﬁguring.
We saw some of this with the package vs source recipes earlier.

Recipes

Avoid hardcoding data
node attributes
data bags
chef search


Chef has a rich set of features that allow us to avoid hardcoding data in recipes

In addition to attributes that we saw earlier, when using the Chef Server we can use data bags
and search.

Code Reuse!

http://www.ﬂickr.com/photos/dnorman/3314634378

Reduce, reuse, recycle.

Design Pattern 4:
Separate recipes


Separate by functionality

default.rb

client.rb

server.rb


default - install common components, should do what one might expect
client - set up to talk to a server, use search to ﬁnd the server based on a role
server - set up the server part, search to ﬁnd clients

Our Nagios cookbook
uses this pattern.

http://www.ﬂickr.com/photos/cote/163746456

%w{
nagios-nrpe-server
nagios-plugins
nagios-plugins-basic
nagios-plugins-standard
}.each do |pkg|
package pkg
end

remote_directory "/usr/lib/nagios/plugins" do
source "plugins"
owner "nagios"
group "nagios"
mode 0755
files_mode 0755
end


nagios::default, packages and plugins.

http://www.ﬂickr.com/photos/zigazou76/3702501888 71

Danger!

We’re going to talk about some Chef Server features.

search(:node, "role:#{node[:nagios][:server_role]}") do |n|
mon_host << n['ipaddress']
end

package "nagios-nrpe-server"

template "/etc/nagios/nrpe.cfg" do
source "nrpe.cfg.erb"
owner "nagios"
group "nagios"
mode "0644"
variables :mon_host => mon_host
notifies :restart, "service[nagios-nrpe-server]"
end

service "nagios-nrpe-server" do
action [:enable,:start]
end


nagios::client is a bit more interesting, where we search for the system that is the monitoring
server and then allow it to connect.

include_recipe "nagios::client"

nodes = search(:node, "hostname:*")

package "nagios3"

template "/etc/nagios3/hosts.cfg" do
source "hosts.cfg.erb"
owner "nagios"
group "nagios"
mode 0644
variables :nodes => nodes
notifies :restart, "service[nagios3]"
end

service "nagios3" do
action [ :enable, :start]
end


Similiarly in nagios::server we search for all the nodes to monitor.

But I use Chef Solo.


So all thats wonderful if you’re using a Server. But you’re not. You’re using Solo.

Beneﬁts of Chef Server

Persistent node data
Arbitrary infrastructure data
Search indexes
API


A sidebar about solo vs server ﬂexibility and reuse you might be missing.

Design Pattern 6L:
Check for Chef Solo.


if Chef::Config[:solo]
node_list = nodes['mything']['node_list']
else
node_list = search(:node, "role:mything")
end


Make a conditional check for solo before doing something like a search, or loading from a
data bag or other server-only feature.

Hardcoding data: Anti-pattern

Customizable cookbooks
Document default attributes
README!

Let users override with roles
Abstract to data bags
non-role/non-node speciﬁc like application info

Play nice with chef-solo


Hardcoding recipes is an anti pattern. We already saw this.

Templates and Files


Lets talk about some of the good things you can do with cookbook assets - templates and
ﬁles

File specificity

cookbooks/mysql/templates
centos/
debian/
default/
redhat/
ubuntu-10.04/
ubuntu-8.04/
ubuntu-9.10/
all contain my.cnf.erb
rendered template picked based on node’s platform


File specificity is useful if your environment has multiple platforms, or if you’re using
cookbooks from others that support platforms different than your own.

Generally, install the package, grab the default config file and stick it in the right directory.

Static vs Dynamic Resources

cookbook_ﬁle is static
template is dynamic

Duh :).


In recipes, use the appropriate resources.

You want dynamic

Easily sharable
Data driven
Rich data from multiple sources


So you want templates. Use them. They’re ERB, so <3 for Rubyists. And they help others
customize your cookbook for their environment with attributes, of course

Libraries, Deﬁnitions,
Resources and
Providers


Cookbooks are more than just recipes and assets (and attributes). Lets talk about libraries,
deﬁnitions, resources and providers.

Libraries

Recipe helpers
LWRP helpers
Heavyweight R/P


Extend chef with libraries, like enhance recipes with helpers.

Recipe Helpers
Use Chef::Recipe class
Methods are available directly in recipes

# in the library
class Chef
class Recipe
def radiant_edge?
node[:radiant][:edge]
end
end
end

# in the recipe
if radiant_edge?
deploy "/srv/radiant" do
repository "git://github.com/radiant/radiant.git"
end
end


Extend the Chef::Recipe class.

LWRP Helpers
Don’t repeat yourself!
Abstract API calls

module Opscode
module Aws
module Ec2
def ec2
@@ec2 ||= RightAws::Ec2.new(
new_resource.aws_access_key,
new_resource.aws_secret_access_key,
{ :logger => Chef::Log }
)
end
end
end
end

# in provider:
include Opscode::Aws::Ec2
...
ec2.describe_addresses.find{|a| a[:public_ip] == ip}
...


in our aws cookbook

Heavyweight Resources & Providers

Full Ruby classes like those in
Chef itself
Allow behaviors not available in
LWRPs
‣ inherit/extend existing resources/provider
Distribute as gems
‣ chef-deploy


Sometimes you might want to write full resources and provides.

Definitions

http://www.flickr.com/photos/thestorylady/4326274437

Don’t use definitions anymore. They look like resources, but they’re actually replaced by the
resources they contain and they don’t send/receive notifications. Instead for more awesome,
use...

Lightweight Resources
& Providers

http://www.ﬂickr.com/photos/lucynieto/2769594798

Aka LWRPs, these are a lightweight DSL for creating new resources and providers in your
cookbooks.

Resource DSL

actions
attributes
validation parameters


Resources really just contain two things, actions and attributes.

Validation Parameters
http://bit.ly/cheﬂwrp

Option Meaning

:default Default value

:kind_of Value must be a kind_of?(Klass)

:required Raise exception if this is missing

:regex Match the value with regular expression

:equal_to Value must match.

:name_attribute Set to the name of the resource.

:callbacks Hash of Procs, should return true.

:respond_to Ensure the value has the given method.


This is on the LWRP page of the wiki.

Example resource

cookbooks/mysql/resources/database.rb

actions :create_db

attribute :host, :kind_of => String
attribute :username, :kind_of => String
attribute :password, :kind_of => String
attribute :database, :kind_of => String
attribute :exists, :default => false


Simple example of a resource. Sorry that they’re called attributes, not the same as node
attributes.

Lightweight Providers

Resources need providers
DSL deﬁnes action methods
Chef Recipe DSL is extended
You can use Chef resources in action methods!


Provider action code
Good!
action :create_db do
unless @mysqldb.exists
Chef::Log.info "Creating database #{new_resource.database}"
execute "create #{new_resource.database}" do
command "mysqladmin -uroot -h localhost create #{new_resource.database}"
end
new_resource.updated_by_last_action(true)
end
end

action :create_db do
Better!
unless @mysqldb.exists
Chef::Log.info "Creating database #{new_resource.database}"
db.query("create database #{new_resource.database}")
new_resource.updated_by_last_action(true)
end
end


Design Pattern 47: Use
moar Ruby!


Reusing resources is cool, but sometimes its better style to use Ruby. It depends. The
advantage of reusing resources is that they’re already idempotent. Except Execute.

Make it idempotent

def load_current_resource
@mysqldb = Chef::Resource::MysqlDatabase.new(new_resource.name)
@mysqldb.database(new_resource.database)
exists = db.list_dbs.include?(new_resource.database)
@mysqldb.exists(exists)
end


Its important to make providers idempotent. The load current resource is called by chef to
see what state the resource is. You have to write the code that determines the state.

Use it in a recipe!

mysql_database "my_app" do
host "localhost"
username "root"
password node['mysql']['server_root_passwd']
database "my_app_production"
action :create_db
end


Metadata


A thing about metadata

Metadata
cookbooks/gnu_parallel/metadata.rb
maintainer "Opscode, Inc."
maintainer_email "cookbooks@opscode.com"
license "Apache 2.0"
description "Installs/Configures gnu_parallel"
long_description IO.read(File.join(File.dirname(__FILE__),
'README.md'))
version "0.99.0"

depends "build-essential"


Declare dependencies on other cookbooks with metadata.

include_recipe


libraries


LWRPs


templates


Anything from other
cookbooks.


Metadata not required
for Chef Solo


because you have to ship all the cookbooks to the node because there’s no chef server to
distribute the required cookbooks

Testing cookbooks


Gotta test that the design was good, right?

No, not BDD/TDD

http://www.ﬂickr.com/photos/davies/4782586685

Though the chef source code itself has a heap of rspec/cucumber tests

Ruby

Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.ﬂickr.com/photos/thisisbossi/3526698689/ 108

Though the code is just ruby, you could write specs and features etc, we don’t have any
examples or know of (m)any people doing this because its easy to test.

knife cookbook upload
(chef-server)


No I mean test it for reals. Upload to the Chef server.

tar -czf cookbooks.tar.gz
(chef-solo)


Or create a tarball for solo.

vagrantup.com


Anyway, you have your cookbooks somewhere the node(s) can get them. Use what you like to
test the recipes. A lot of people like Vagrant.

knife ec2 server create


Some people like ec2. Or rackspace, whatever.

NOT

http://www.ﬂickr.com/photos/valeriebb/290711738 113

Use what you like for test machines. That part isn’t important.

sudo chef-client

sudo chef-solo

Whatever, you need to run Chef to test.

http://www.ﬂickr.com/photos/billburris/2245430380/ 115

Show your work

git push


Put it on your github repository!

knife cookbook site share


go that extra step and publish on the cookbooks site.

ﬂat namespace like rubygems.org though but we’re working to change that


Posted it to the cookbooks site.

Opscode’s cookbook examples

aws
gnu_parallel
mysql
nagios
radiant

http://ckbk.it/NAME


Thanks!

www.opscode.com/chef
IRC and Mailing lists
‣ irc.freenode.net #chef
‣ lists.opscode.com

Twitter:
‣ @opscode, #opschef
‣ @jtimberman


Mwrc2011 cookbook design patterns

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Mwrc2011 cookbook design patterns

Ähnlich wie Mwrc2011 cookbook design patterns (20)

Mwrc2011 cookbook design patterns