3. Who am I?
• Seth Chisamore
Senior Technical Consultant/Evangelist
Opscode, Inc.
4. Who am I?
• Seth Chisamore
Senior Technical Consultant/Evangelist
Opscode, Inc.
• Project lead for many Opscode sponsored
open source projects:
• Opscode Cookbooks
• Knife cloud provisioning plugins (ec2,
rackspace & openstack)
• Knife Windows plugin (knife-windows)
5. Who are you?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
6. Who are you?
• System administrators?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
7. Who are you?
• System administrators?
• Developers?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
8. Who are you?
• System administrators?
• Developers?
• “Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
9. What are we talking
about?
http://www.flickr.com/photos/peterkaminski/2174679908/
11. Agenda
• How’s and Why’s
http://www.flickr.com/photos/koalazymonkey/3590953001/
12. Agenda
• How’s and Why’s
• Chef 101
http://www.flickr.com/photos/koalazymonkey/3590953001/
13. Agenda
• How’s and Why’s
• Chef 101
• Deploying n-tier Django app stacks
with Chef
http://www.flickr.com/photos/koalazymonkey/3590953001/
14. Agenda
• How’s and Why’s
• Chef 101
• Deploying n-tier Django app stacks
with Chef
• Leveraging the same Chef code to
deploy different configurations of
our stack
http://www.flickr.com/photos/koalazymonkey/3590953001/
49. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
50. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
51. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
• 240+ cookbooks
52. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
• 240+ cookbooks
• http://community.opscode.com
59. Chef Enables Infrastructure as Code
package "haproxy" do
action :install
end
• Resources template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Recipes owner "root"
group "root"
• Cookbooks mode 0644
•
notifies :restart, "service[haproxy]"
Roles end
• Data Bags service "haproxy" do
• Source Code supports :restart => true
action [:enable, :start]
end
61. Chef Resources
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end
62. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end
63. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end
64. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end
65. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
• Take action to put the resource notifies :restart, "service[haproxy]"
end
in the declared state.
service "haproxy" do
supports :restart => true
action [:enable, :start]
end
66. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
• Take action to put the resource notifies :restart, "service[haproxy]"
end
in the declared state.
• Can send notifications to other service "haproxy" do
supports :restart => true
resources. action [:enable, :start]
end
68. Chef Providers
• Multiple providers per resource type.
• For example the `package` resource
has providers for: Apt, Yum, Rubygems,
Portage, Macports, FreeBSD Ports, etc.
• If you are developer think of it as an
interface and an implementation.
70. Chef Recipes
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Recipes are evaluated for owner "root"
resources in the order they group "root"
mode 0644
appear. notifies :restart, "service[haproxy]"
• Each resource object is added end
to the Resource Collection. service "haproxy" do
supports :restart => true
action [:enable, :start]
end
71. Chef Recipes
• Recipes can include other include_recipe
include_recipe
"apache2"
"apache2::mod_rewrite"
recipes. include_recipe "apache2::mod_deflate"
• Included recipes are
include_recipe
include_recipe
"apache2::mod_headers"
"apache2::mod_php5"
processed in order.
72. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
pool_members = search("node", "role:django_cms")
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end
73. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
• Extend recipes with
pool_members = search("node", "role:django_cms")
Ruby.
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end
74. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
• Extend recipes with
pool_members = search("node", "role:django_cms")
Ruby.
• Dynamic configuration template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
through search. owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end
90. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
91. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:
92. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:
• many app servers to checkout the same
application code during initial
bootstrapping
93. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:
• many app servers to checkout the same
application code during initial
bootstrapping
• app server and database servers to
initialize off of the same database config
118. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.
119. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.
• physical hardware or any number of
public/private cloud providers.
120. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.
• physical hardware or any number of
public/private cloud providers.
• a developer workstation fully virtualized
with Vagrant + VirtualBox
121. Staging/QA - Single Node
# launch stack in your data center
% knife bootstrap 123.123.3.3
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’
# launch stack in EC2
% knife ec2 server create
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’
# launch stack in Rackspace
% knife rackspace server create
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’
122. Production - Multi-Machine Cluster
% knife ec2 server create
-r ‘role[django_cms_database_master]’
% knife ec2 server create
-r ‘role[django_cms]’
% knife ec2 server create
-r ‘role[django_cms]’
% knife ec2 server create
-r ‘role[django_cms_load_balancer]’
123. Developer Workstation
% cat Vagrantfile
Vagrant::Config.run do |config|
config.vm.box = base_box
config.vm.provision :chef_server do |chef|
chef.chef_server_url = "https://api.opscode.com/organizations/foo"
chef.validation_key_path = "~/.chef/private-pems/foo-validator.pem"
chef.validation_client_name = "#{ENV['ORGNAME']}-validator"
chef.provisioning_path = "/etc/chef"
chef.run_list = [
'role[django_cms_database_master]',
'role[django_cms]',
'role[django_cms_load_balancer]'
]
end
end
% vagrant up
125. Application Cookbook
• Supports the following web stacks:
• Django
• Java Webapps
• Php
• Rails
• Node.js (coming soon)
• Opscode contributed quick starts walk you
through example deployments
How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
Why not just use some shell scripts?\n
\n
This is the dream. When everyone’s 2nd girlfriend ‘nagios’ wakes you up in the middle of the night you are ready for the worst.\n
Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure.\n\nIt is all about the policy. This can be lo-fi (ie a wiki page and the ‘meat’ cloud) or fully automated with something like Chef.\n\nUsually this is the context of a single node.\n
Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.\n\nWRT Chef, we talk about Fully Automated Infrastructure.\n\nThis brings all the single nodes together.\n
Chef provides a framework for fully automating infrastructure, and has some important design principles.\n
Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
Idempotent: Consistency, known system state, easy to build\nChef takes idempotent actions to put the system in the desired state.\n
Chef makes it easy to reason about your infrastructure.\n\nWe build an API, with primitives and libraries, and let you model your data. We don’t try to discover because we might make an incorrect assumption about the system.\n\nYou need an API to manage your infrastructure. The standard tools on Unix and Windows already use APIs to the system. Use an API to manage your infrastructure.\n
Chef comes from the perspective that the tool should do pretty much what you expect, hence sane defaults. To get there we had express some opinions. Those opinions can be changed easily, whether that is modifying the configuration of the chef-client, or how Apache is set up.\n
Chef is a flexible framework. You can make it do pretty much anything you want. Just like Perl doesn’t tell the programmer how to program, Chef doesn’t tell the system administrator how to manage the system. You can extend it, mold it and modify it to do what you need.\n\nBeing written in Ruby, using Ruby for the main language was very purposeful. The tool shouldn’t get in your way.\n
\n
-Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
-Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
-Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
-Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
\n
-thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
-thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
-thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
-thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
\n
\n
Communication with the server is secure. Clients hold the private key, the server has the public key. The server is not a centralized store of authentication information, pushing the responsibility to the nodes.\n
\n
\n
\n
Manage system configuration as resources.\nPut resources together in recipes.\nDistribute recipes with cookbooks.\nAssign recipes to systems through roles.\nStore arbitrary data in data bags.\nTrack it all like source code.\n
Resources are an abstraction we feed data into. When you write recipes in Chef, you create resources of things you want to configure.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
Resources declare a description of the state a part of the node should be in.\n
The abstraction over the commands or API calls that will configure the resource to be in the state you have defined.\n
Providers know how to actually configure the resources to be in the declared state\n\n
\n
\n
Just like recipes themselves are processed in order, the recipes included are processed in order, so when you include a recipe, all its resources are added to the resource collection, then Chef continues to the next.\n\n\n
\n
\n
\n
\n
\n
\n
\n
-the cpan, PyPI or rubygems.org of the cookbbook world!\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Data bags are containers of grouped information.\n\nUsers, application information, network info, cabinet/rack locations. Describe components of your infrastructure with data, and use that data to configure systems.\n\nProcess data bags in recipes to configure systems dynamically.\n
\n
\n
In our example we are deploy Django CMS but this would apply to any Django application.\n
Provisioning is the first step. We need some computers on the internet. They’re going to be load balancers, webservers, database servers. Launch those with a cloud API. Every cloud does this.\n
You need to do something and you manage the configuration. Install packages, create users, etc. You can use pre-built images for this, but then just for a simple infrastructure (w/o monitoring/trending, middleware, etc) you have four discrete images to maintain for updates both for the OS and app updates.\n
This is the last mile. The systems are configured, but also need to be configured to talk to each other.\n
\n
\n
\n
\n
Every database backed application needs a master database. For this simple example we haven’t done any complex setup of master/slave replication, but the recipes are built such that this would be relatively easy to add.\n
The database master recipe will read the application information from the data bag and use it to create the database so the application can store its data.\n
all done fully automated\n
all done fully automated\n
all done fully automated\n
We start with one but scale horizontally as needed. We can even scale up and down based on demands.\n
The main thing in this role is the application recipe. \n\nThe recipe will read in data from the data bag (in a predefined format) to determine what kind of application to deploy, the repository where it lives, details on where to put it, what roles to search for to find the database, and many more customizable properties.\n\nWe launched two of these to have something to load balance :).\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
-acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
You no longer need to track which system has an IP that should be applied as the database master. We can just use its fqdn from a search.\n
\n
We’re using haproxy, and we’ll search for a specific application to load balance. The recipe is written to search for the django_cms role to find systems that should be pool members.\n
\n
-This search occurs everytime Chef runs on the load balancer.\n-If you add more app servers to the mix, the load balancer will *automatically* become aware!\n
A single set of Chef cookbooks, roles and databags allows you to deploy your application stack in numerous ways.\n