Presentation given at PDX Cloud meeting on Chef

1. Configuration Management in the Cloud
with Chef
PDX Cloud
James Casey
james@opscode.com
Twitter: jamesc_000
GitHub: jamesc
www.opscode.com
Tuesday, April 30, 13

2. • Instant infrastructure
• Unlimited capacity
• Autoscaling
• No commitment
• Immediate replacement
Clouds are great !
Tuesday, April 30, 13

3. APIs are awesome!
• You can provision
compute resources in
seconds
• You can provision
storage resources in
seconds
• You can provision
network resources in
seconds
• That’s cool.
http://www.flickr.com/photos/jdhancock/3634246981/
Tuesday, April 30, 13

4. The Dark Side
of the Cloud
Tuesday, April 30, 13

5. • Performance
• Security
• Price
• Reliability
Why not the Cloud?
Tuesday, April 30, 13

6. See Node
Application Server
Tuesday, April 30, 13

7. See Nodes
Application Server
Application Database
Tuesday, April 30, 13

8. See Nodes Grow
Application Server
Application Databases
Tuesday, April 30, 13

9. Application Servers
Application Databases
See Nodes Grow
Tuesday, April 30, 13

10. Application Servers
Application Databases
Load Balancer
See Nodes Grow
Tuesday, April 30, 13

11. See Nodes Grow
Application Servers
Application Databases
Load Balancers
Tuesday, April 30, 13

12. See Nodes Grow
Application Servers
Application Database Cache
Load Balancers
Application Databases
Tuesday, April 30, 13

13. Tied together with Config
Application Servers
Application Database Cache
Load Balancers
Application Databases
Tuesday, April 30, 13

14. Infrastructure is a Snowflake
Application Servers
Application Database Cache
Load Balancers
Floating IP?
Application Databases
Tuesday, April 30, 13

15. Evolving Complexity
Load Balancers
Application Servers
NoSQL
Database Slaves
ApplicationCache
Database Cache
Database
Tuesday, April 30, 13

16. Complexity Grows Quickly
DC1
DC3
DC2
Tuesday, April 30, 13

17. http://www.flickr.com/photos/16339684@N00/2681435235/
And it Continues to Evolve
Tuesday, April 30, 13

18. http://www.flickr.com/photos/16339684@N00/2681435235/
And it Continues to Evolve
Ok, so I’ve got a problem.
What’s the solution ?
Tuesday, April 30, 13

19. Golden Images are not the answer
• Gold is heavy
• Hard to transport
• Hard to mold
• Easy to lose
configuration detail
http://www.flickr.com/photos/garysoup/2977173063/
Tuesday, April 30, 13

20. Configuration Management and
Automated Systems Integration
is the Answer
http://www.flickr.com/photos/philliecasablanca/3354734116/
Tuesday, April 30, 13

21. • Turn code and hardware into
infrastructure
• From bare metal to services in
production
• Scale applications as needed
• Conform to policy
• Align to business goals
Configuration Management ?
Tuesday, April 30, 13

22. Chef - Infrastructure as Code
http://www.flickr.com/photos/louisb/4555295187/
• Programmatically
provision and configure
• Treat like any other code
base
• Reconstruct business from
code repository, data
backup, and bare metal
resources.
Tuesday, April 30, 13

23. • Chef-Client generates
configurations directly
on nodes from their
run list
• Reduce management
complexity through
abstraction
• Store the configuration
of your programs in
version control
http://www.flickr.com/photos/ssoosay/5126146763/
Nodes
Tuesday, April 30, 13

24. Collections of Resources
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Routes
• Users
• Groups
• Tasks
• Packages
• Software
• Services
• Configurations
• Other Stuff
http://www.flickr.com/photos/stevekeys/3123167585/
Tuesday, April 30, 13

25. Declarative Interface to Resources
• Define policy
• Say what, not how
• Pull not Push
http://www.flickr.com/photos/bixentro/2591838509/
Tuesday, April 30, 13

26. Ruby!
extra_packages = case node['platform']
when "ubuntu","debian"
%w{
ruby1.8
ruby1.8-dev
rdoc1.8
ri1.8
libopenssl-ruby
}
end
extra_packages.each do |pkg|
package pkg do
action :install
end
end
Tuesday, April 30, 13

27. Or this
search(:users, '*:*') do |u|
user u['id'] do
uid u['uid']
shell u['shell']
home "/home/#{u['id']}"
end
directory "#{home_dir}/.ssh" do
owner u['id']
group u['gid']
mode "0700"
end
template "#{home_dir}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode "0600"
variables :ssh_keys => u['ssh_keys']
end
end
Tuesday, April 30, 13

28. Recipes and Cookbooks
• Recipes are collections of
Resources
• Cookbooks contain
recipes, templates, files,
custom resources, etc
• Code re-use and
modularity
• Hundreds already on
Community.opscode.com
http://www.flickr.com/photos/shutterhacks/4474421855/
Tuesday, April 30, 13

29. http://www.flickr.com/photos/kathycsus/2686772625
• IP addresses
• Hostnames
• FQDNs
• Search for nodes
with Roles
• Find configuration
data
Search
Tuesday, April 30, 13

30. pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members.uniq
notifies :restart, "service[haproxy]"
end
Pass Results to Templates
Tuesday, April 30, 13

31. # Set up application listeners here.
listen application 0.0.0.0:80
balance roundrobin
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
<% if node["haproxy"]["enable_admin"] -%>
listen admin 0.0.0.0:22002
mode http
stats uri /
<% end -%>
Pass Results to Templates
Tuesday, April 30, 13

32. Jboss App
Memcache
Postgres Slaves
Postgres Master
So when this
NagiosGraphite
Tuesday, April 30, 13

33. Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Becomes this
Tuesday, April 30, 13

34. Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Updates can be automatic
Tuesday, April 30, 13

35. NagiosGraphite
Count the resources
Jboss App
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
• 12+ resource changes for 1 node addition
Tuesday, April 30, 13

36. http://www.flickr.com/photos/evelynishere/2798236471/
CLONING CANNOT COPE WITH THIS
• Chef can.
Tuesday, April 30, 13

37. Build anything
• Simple internal applications
• Complex external applications
• Workstations
• Hadoop clusters
• IaaS infrastructure
• PaaS infrastructure
• SaaS applications
• Storage systems
• You name it
http://www.flickr.com/photos/hyku/245010680/
Tuesday, April 30, 13

38. And manage it simply
http://www.flickr.com/photos/helico/404640681/
• Automatically
reconfigure
everything
• Linux,Windows,
Unixes, BSDs
• Load balancers
• Metrics collection
systems
• Monitoring systems
• Cloud migrations
become trivial
Tuesday, April 30, 13

39. knife
Tuesday, April 30, 13

40. Upload your infrastructure
knife cookbook upload apt
knife cookbook upload chef-client
knife cookbook upload java
knife cookbook upload jpackage
knife cookbook upload ntp
knife cookbook upload sudo
knife cookbook upload tomcat
knife cookbook upload users
knife cookbook upload sample
knife role from file base-cloud.rb
knife role from file tc.rb
knife role from file sample.rb
knife data bag create users
knife data bag from file users jamesc.json
Tuesday, April 30, 13

41. Build it somewhere
#EC2
knife ec2 server create -S jamesc -i ~/.ssh/jamesc.pem -x ubuntu -G default -I ami-
a7a97dce -f m1.small -d omnibus -r 'role[base-cloud],role[tc],role[sample]'
#Rackspace
knife rackspace server create --image 110 --flavor 2 -i ~/.ssh/jamesc.pem -d omnibus -r
'role[base-cloud],role[tc],role[sample]'
#CloudStack
knife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i
~/.ssh/jamesc.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]'
#Ubuntu Linux
knife bootstrap test.lab -r 'role[webserver]' -i ~/.ssh/jamesc.pem -x ubuntu --sudo -d
omnibus -r 'role[base-cloud],role[tc],role[sample]'
Tuesday, April 30, 13

42. knife ec2
$ knife ec2
Available ec2 subcommands: (for details, knife SUB-COMMAND --
help)
** EC2 COMMANDS **
knife ec2 flavor list (options)
knife ec2 instance data (options)
knife ec2 server create (options)
knife ec2 server delete SERVER [SERVER] (options)
knife ec2 server list (options)
$ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu
-I ami-4721882e -f m1.small -r 'role[webserver]'
Tuesday, April 30, 13

43. knife openstack
$ knife openstack
Available openstack subcommands: (for details, knife SUB-
COMMAND --help)
** OPENSTACK COMMANDS **
knife openstack flavor list (options)
knife openstack image list (options)
knife openstack server create (options)
knife openstack server delete SERVER [SERVER] (options)
knife openstack server list (options)
$ knife openstack server create -S keypair -i ~/.ssh/id_rsa
-x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
Tuesday, April 30, 13

44. Chef for Infrastructure Portability
• knife ec2
• knife rackspace
• knife hp
• knife google
• knife azure
• knife cloudstack
• knife openstack
• knife vsphere
• ... and many
others
Tuesday, April 30, 13

45. The Chef Community
• Apache License,Version 2.0
• 850+ Individual contributors
• 150+ Corporate contributors
• HP, Dell, Rackspace,VMware, Joyent,
Calxeda, Heroku, SUSE and many more
• 550+ cookbooks
• http://community.opscode.com
Tuesday, April 30, 13

46. Summary
• Every infrastructure is a unique snowflake
• You need tools to let you do what you
want
• You need the power to grow your
infrastructure
• You need the ability to change your
cloud provider
• Automated Configuration Management is
the solution
Tuesday, April 30, 13

47. Questions?
http://www.flickr.com/photos/mrchippy/443960682/
Questions?
Tuesday, April 30, 13

48. Thanks!
James Casey
james@opscode.com
Twitter: jamesc_000
Github: jamesc
www.opscode.com
Tuesday, April 30, 13