Prepping the Kitchen : Chef Concepts and Fundamentals sides used at Surge 2011

2. Prepping the Kitchen - Chef Concepts
and Fundamentals
someara@opscode.com
www.opscode.com

3. Overview
• Infrastructure as code
• Configuration Management
Strategies
• Chef

4. Infrastructure as code

5. Infrastructure
"It is common to think in terms of individual machines
rather than view an entire infrastructure as a
combined whole"
“A good infrastructure, whether departmental,
divisional, or enterprise-wide, is a single loosely-
coupled virtual machine, with hundreds or
thousands of hard drives and CPU's.”
-- Bootstrapping an Infrastructure USENIX LISA ’98
http://www.infrastructures.org/papers/bootstrap/bootstrap.html

6. .... as code!
• Programmatically provision and
configure
• Treat like any other code base
• Reconstruct operations from code
repository, data backup, and bare
metal resources.
http://www.flickr.com/photos/louisb/4555295187/

7. Considerations
• Infrastructure changes over time
• Entropy
• Changing business requirements
http://www.flickr.com/photos/seatbelt67/502255276/

8. Methodology
http://www.flickr.com/photos/drachmann/327122302/

9. Configuration Management
Strategies

10. Manual
Configuration
• Labor intensive
• Error prone
• Hard to reproduce
• Unsustainable
http://www.flickr.com/photos/pureimaginations/4805330106/

11. Scripting
• Typically very brittle
• Throw away, one off scripts
• grep sed awk perl
• curl | bash
http://www.flickr.com/photos/40389360@N00/2428706650/

12. File
Distribution
• NFS mounts
• rdist
• scp-on-a-for-loop
• rsync on cron
http://www.flickr.com/photos/walkadog/4317655660

13. This used to be
awesome
for i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/
ntpd.conf ; done
for i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpd
restart ; done
for i in `cat servers.txt` ; do ssh root@$i chkconfig ntpd
on ; done
• ^ does not scale
http://www.flickr.com/photos/alexerde/3479006495

14. Declarative
Syntax
• Define policy
• Say what, not how
• Abstract interface to
resources
• Enables some interesting
behavior
http://www.flickr.com/photos/bixentro/2591838509/

15. Declarative Syntax
Declarative Tools
• LCFG
• CFEngine
• BCFG2
• Puppet
• Chef

16. Declarative Syntax
package "ntp" do
action :install
Idempotence
•
end
cookbook_file "/etc/ntp.conf" do
You’ll hear this a lot
source "ntp.conf"
owner "root" • Property of declarative
group "root" interface
mode 0644
action :create
notifies :restart, “service[ntpd]”
• Eliminates brittleness of
end
scripting
service "ntpd" do
action [:enable,:start]
• Identity function: f(x)=x
end

17. Declarative Syntax
while true do
package "ntp" do
action :install
Idempotence
•
end
cookbook_file "/etc/ntp.conf" do
You’ll hear this a lot
source "ntp.conf"
owner "root" • Property of declarative
group "root" interface
mode 0644
action :create
notifies :restart, “service[ntpd]”
• Eliminates brittleness of
end
scripting
service "ntpd" do
action [:enable,:start]
• Identity function: f(x)=x
end • Safe to repeat
end

18. Declarative Syntax
Convergence
• Agents “converge” a system to
desired state
• Repetition inches closer to
desired state
• It eventually gets there
• SCIENCE!
http://www.flickr.com/photos/tolomea/4852616645/

19. Declarative Syntax
Convergence
service "ntpd" do
action [:enable,:start]
ignore_failure true
end
cookbook_file "/etc/ntp.conf" do • Agents “converge” a system to
source "ntp.conf" desired state
owner "root"
group "root"
mode 0644 • Repetition inches closer to
action :create desired state
•
notifies :restart, “service[ntpd]”
ignore_failure true It eventually gets there
•
end
SCIENCE!
package "ntp" do
action :install
ignore_failure true
end

20. Declarative Syntax
# echo “boom” > /etc/ntp.conf ;
chef-client
Convergence
$ grep server /etc/ntp.conf | head -n 1
us.pool.ntp.org
$ ps -e | grep ntp
• Fights entropy, unauthorized
1799 ? 00:00:00 ntpd
changes, and gingivitis
# /etc/init.d/ntpd stop ; chef-client • Update function inputs to deal
with changing requirements
ps -e | grep ntp
1822 ? 00:00:00 ntpd

21. Config Generation
• Often made by hand (still!?)
• Stop that.
• Generate them based on
database content
• Infrastructures evolve
http://www.flickr.com/photos/jabella/4753170413/

22. See Node
Application

23. See Nodes
Application
Application Database

24. See Nodes Grow
Application
App Databases

25. See Nodes Grow
App Servers
App Databases

26. See Nodes Grow
App LB
App Servers
App Databases

27. See Nodes Grow
App LBs
App Servers
App Databases

28. See Nodes Grow
App LBs
App Servers
App DB Cache
App DBs

29. Stitched together with configs
App LBs
App Servers
App DB Cache
App DBs

30. Stitched together with configs
App LBs
App Servers
App DB Cache
Floating IP?
App DBs

31. Complexity increases quickly
App LBs
Cache
App Servers
NoSQL DB Cache
DB slaves
DBs

32. Complexity increases very quickly
DC2
DC1
DC3

33. Generate configs
• Centralized generation
• Version control!
• Distribute with packages, Chef,
git, whatever.
http://www.flickr.com/photos/ssoosay/5126146763/

34. Generate configs
• Local generation directly on nodes
• Reduces management complexity
• No need to distribute
• Version control the programs instead
http://www.flickr.com/photos/ssoosay/5126146763/

35. Chef

36. All That Stuff
• Declarative interface to resources
• Database of nodes and their roles
• Grab remote configs
• Generate configs locally

37. and more!
• Data Driven Infrastructure
• Use APIs to obtain data
• chef-server, SQL, anything.
• Feed resources parameters
• IPs, FQDNs, memory sizes,
• Templates, package, firewall
rules

38. Architecture
• Code Repository
• Chef Server
• Chef Clients
• Data Bags
• Recipes and Cookbooks
• Roles and Run Lists
http://www.flickr.com/photos/boedker/3871267007

39. Code Repository
• Version control
• Development
workflows
• Sharing is Caring

40. Chef Server
Server
Server
chef-server
Server
Server
• Upload from laptop
with knife
RESTful API
Cookbook
Cookbook
Cookbook
Data Bag
Knife Knife
Role
Knife

41. Chef Clients
Server
Server
chef-server
Server
Server
Knife
• Clients are API users
• Read RESTful API Knife
• Write
• Search
chef-client
chef-client chef-client chef-client chef-client

42. Chef Clients
Server
Server
someara.pub chef-server
Server
Server
jtimberman.pub
node5.fqdn.pub
Knife someara.pem
• Clients are API users
• Public keys on server RESTful API Knife jtimberman.pem
• Private keys local to
machines
chef-client
chef-client chef-client chef-client chef-client
node5.fqdn.pem

43. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
API chef-client
Give me
recipe[ntp::client]
ntp
node
client.rb

44. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
chef-client
API Give me
“ntp::client”,
ntp “openssh::server”
openssh
node
client.rb
server.rb

45. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
chef-client Give me
API
“recipe[ntp::client]”,
ntp “recipe[openssh::server]”,
“recipe[apache]”,
openssh
node “recipe[php]”
client.rb apache
server.rb php
default.rb
default.rb

46. Roles
Server
Server
chef-server
Server
Server
Role Recipe
API
Role
Role Recipe
Role Recipe
Recipe
Knife
Recipe
Recipe
Recipe

47. Roles
Server
Server
chef-server
Server
Server
chef-client Ohai!
API
Give me
ntp “role[base]”,
“role[webserver]”
openssh
node
client.rb apache
server.rb php
default.rb
default.rb

48. Roles
Server
Server
chef-server
Server
Server
ntp
openssh
chef-client
API client.rb apache
php
server.rb “role[webserver]”
default.rb
ntp default.rb node
openssh
chef-client
client.rb mysql
server.rb
server.rb
“role[database]”
node

49. Bootstrapping
nodes
• Get chef-client installed
• Write run list to a file
• “Press go”
http://www.flickr.com/photos/liftarn/1447521121/

50. Bootstrapping nodes
• knife ec2 server create -r
‘role[webserver]’
• knife bootstrap 10.9.8.7 -r
‘role[webserver]’
• Cobbler
http://www.flickr.com/photos/hakonjarl/4010080214/

51. Bootstrapping nodes
{
"kernel": {
"machine": "x86_64",
"name": "Darwin",
"os": "Darwin",
"version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;
•
root:xnu-1504.7.4~1/RELEASE_I386",
Ohai generates a JSON },
"release": "10.4.0"
attributes list "platform_version": "10.6.4",
•
"platform": "mac_os_x",
Run list and attributes are "platform_build": "10F569",
"domain": "local",
combined into a Node object "os": "darwin",
"current_user": "mray",
•
"ohai_time": 1278602661.60043,
Can be viewed and "os_version": "10.4.0",
"uptime": "18 days 17 hours 49 minutes 18 seconds",
searched through API "ipaddress": "10.13.37.116",
"hostname": "morbo",
"fqdn": "morbomorbo.local",
"uptime_seconds": 1619358
}

52. Bootstrapping nodes
• Run list is requested
• Cookbooks downloaded
• Recipes executed
• Node saved to chef-server
http://www.flickr.com/photos/architopher/457885721

53. Cookbooks
and Recipes
• Cookbooks contain recipes
• And everything they need to
work
• Templates, files, custom
resources, etc
http://www.flickr.com/photos/shutterhacks/4474421855/

54. Cookbooks
$ tree -a cookbooks/haproxy/
README.md
attributes
default.rb
• Cookbooks contain recipes metadata.rb
• And everything they need to recipes
work app_lb.rb
default.rb
• Templates, files, custom templates
resources, etc default
haproxy-app_lb.cfg.erb
haproxy-default.erb
haproxy.cfg.erb

55. Recipes
package "haproxy" do
action :install
end
template "/etc/default/haproxy" do
source "haproxy-default.erb"
• Recipes contain lists of owner "root"
group "root"
resources mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
action [:enable, :start]
end

56. Resources

57. Resources
package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end

58. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end

59. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end

60. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
•
template "/etc/apache2/apache2.conf" do
Have parameters source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end

61. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
•
template "/etc/apache2/apache2.conf" do
Have parameters source "apache2.conf.erb"
owner "root"
• Take action to put the resource group "root"
mode 0644
in the declared state action :create
end

62. Searching
http://www.flickr.com/photos/fotos_medem/3399096196/

63. Searching
• All object in Chef server are
indexed by Solr
http://www.flickr.com/photos/fotos_medem/3399096196/

64. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
http://www.flickr.com/photos/fotos_medem/3399096196/

65. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
• From knife and in recipes
http://www.flickr.com/photos/fotos_medem/3399096196/

66. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
• From knife and in recipes
• Returns an array of JSON Node
objects
http://www.flickr.com/photos/fotos_medem/3399096196/

67. Systems Integration
knife search node role:webserver
webservers = search("node", "role:webserver”)

68. Pass results into Templates
pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members.uniq
notifies :restart, "service[haproxy]"
end

69. Pass results into Templates
pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members.uniq
notifies :restart, "service[haproxy]"
end

70. Pass results into Templates
# Set up application listeners here.
listen application 0.0.0.0:80
balance roundrobin
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1
check
<% end -%>
<% if node["haproxy"]["enable_admin"] -%>
listen admin 0.0.0.0:22002
mode http
stats uri /
<% end -%>

71. Change
• Various ways
• Add or remove a node to the
infrastructure
• Run chef-client

72. Run chef-client
$ grep servers /etc/haproxy/haproxy.cfg
servers node2.mylan 10.9.8.10
servers node3.mylan 10.9.8.11
$ knife ec2 server create -r ‘webserver’
$ knife ec2 server create -r ‘webserver’
$ knife ssh ‘role:webserver’ chef-client
$ grep servers /etc/haproxy/haproxy.cfg
servers node2.mylan 10.9.8.10
servers node3.mylan 10.9.8.11
servers node4.mylan 10.9.8.12
servers node5.mylan 10.9.8.13

73. Change Inputs
• Edit recipes
• Edit run lists
• chef-client
http://www.flickr.com/photos/dhutchman/128541987

74. Out of slides!
http://www.flickr.com/photos/calonyr11/2630312566/

75. Questions?
sales@opscode.com
www.opscode.com