Regardless of whether you're using chef or any other automated devops tool, you still need to consider where you are going to host things. Redundancy is good, so in this talk I will describe the tools I used as well as how and why I set up my own chef+git server to provide my own cauldron in which to cook up server deployments.
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
Building a Cauldron for Chef to Cook In
1. Building a Cauldron for Chef to
Cook In
Jonathan Altman
@async_io
For DevOpsDC August 2013
Wednesday, August 14, 2013
2. Background
• 12 years architecting at a leading SaaS software provider
• Started 2 companies in August 2012
• One is a SaaS/product based company
• One is a services firm
• Customers have access to their source
• No time to waste: automation good
Wednesday, August 14, 2013
3. What is a Cauldron?
• A complete (as you want) code/artifact management and build/deployment system
• Chef server
• Git server
• Automated build system e.g. Jenkins (eventually)
• JIRA server?
• Group chat server? (IRC or XMPP)
• With a real cert
Wednesday, August 14, 2013
4. But SaaS is Awesome!
• Yes. Yes it is.
• Github
• Hosted Chef
• ShiningPanda, CloudBees, JenkinsHosting, travis-ci.com
• Jira OnDemand
• HipChat (or https://grove.io/ for hosted IRC, or Google+ hangouts)
Wednesday, August 14, 2013
5. The 3 R’s
• Redundancy
• Resiliency
• Revelation
Wednesday, August 14, 2013
6. Redundancy
• Does a good backup strategy ever keep just one copy of something?
• Disaster Recovery: how do you recover with your backups?
• At least for git, chef, and CI you can use multiple servers
• git “just works” in this model
• The knife plugin has backup/restore capability for example
• Not sure how to integrate with Opscode-hosted chef
Wednesday, August 14, 2013
7. Resiliency
• Tradeoff between your ability to deliver uptime and their motivation
to address their issues that affect your uptime
• Honeypot: tradeoff again. github and hosted chef server are well
protected, but a big, attractive attack vector. Your server, not so
much; but is it secure?
• Ability to assign the amount of compute resources you need to
deliver the performance, uptime, and redundancy you want
Wednesday, August 14, 2013
8. Revelation
• You will have the tools and ability to help yourself
• Git != Github, for example
Wednesday, August 14, 2013
9. A Cauldron: workstation
• Client workstation with
• omnibus installer (http://www.opscode.com/chef/install/)
• knife-server plugin (http://fnichol.github.io/knife-server/)
• git client software installed on it
• You could do this on your workstation’s host OS, but using aVM
simplifies/isolates having multiple cauldrons
Wednesday, August 14, 2013
10. Create Cauldron Workstation
Cocytus:vagrant_servers jonathan$ mkdir cauldron_ws
Cocytus:vagrant_servers jonathan$ cd cauldron_ws
Cocytus:cauldron_ws jonathan$ vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
Cocytus:cauldron_ws jonathan$
Wednesday, August 14, 2013
11. Bootstrap theVagrant
Cocytus:cauldron_ws jonathan$ vi Vagrantfile # Set up your config here
Cocytus:cauldron_ws jonathan$ vagrant up
[default] Box base was not found. Fetching box from specified URL...
[vagrant] Downloading with Vagrant::Downloaders::HTTP...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...
[default] Importing base box 'base'...
{bunch of lines deleted ...}
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
VM must be created before running this command. Run `vagrant up` first.
Cocytus:cauldron_ws jonathan$
Wednesday, August 14, 2013
12. Install chef omnibus
Cocytus:cauldron_ws jonathan$ vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Welcome to your Vagrant-built virtual machine.
Last login: Fri Sep 14 06:23:18 2012 from 10.0.2.2
vagrant@precise64:~$ sudo apt-get install build-essential curl
[...bunch of output deleted]
vagrant@precise64:~$ curl -L https://www.opscode.com/chef/install.sh | sudo bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6790 100 6790 0 0 22442 0 --:--:-- --:--:-- --:--:-- 36902
Downloading Chef for ubuntu...
Installing Chef
Selecting previously unselected package chef.
(Reading database ... 51127 files and directories currently installed.)
Unpacking chef (from .../tmp.8PmNsIUQ/chef__amd64.deb) ...
Setting up chef (11.6.0-1.ubuntu.12.04) ...
Thank you for installing Chef!
vagrant@precise64:~$
Wednesday, August 14, 2013
13. Knife-server plugin
• knife plugin to create chef servers on the command line running on
ubuntu:
• on ec2 (requires the knife-ec2 plugin as well)
• on linode (requires the knife-linode plugin)
• on an already-running instance (“standalone”)
Wednesday, August 14, 2013
14. Install knife-server Plugin
vagrant@precise64:~$ sudo /opt/chef/embedded/bin/gem install knife-server
--no-ri --no-rdoc
Building native extensions. This could take a while...
Fetching: ruby-hmac-0.4.0.gem (100%)
Fetching: fog-1.14.0.gem (100%)
Fetching: knife-server-1.1.0.gem (100%)
Successfully installed nokogiri-1.6.0
Successfully installed ruby-hmac-0.4.0
Successfully installed fog-1.14.0
Successfully installed knife-server-1.1.0
4 gems installed
vagrant@precise64:~$
Wednesday, August 14, 2013
15. Make a Server (Standalone)
• Standalone there is a bunch of other stuff to get right:
• Better if the server you’re installing on has root with ssh
authorized_keys set up to your workstation’s ssh private key
• Need both private and public part of key on workstation because of
http://tickets.opscode.com/browse/CHEF-4180
Wednesday, August 14, 2013
16. Server Bootstrapping...
knife server bootstrap standalone --node-name cauldron.your.domain --host
xxx.yyy.zzz.aaa
[hundreds of lines of output deleted ...]
192.241.179.65 Recipe: chef-server::erchef
192.241.179.65 * service[erchef] action restart
192.241.179.65
192.241.179.65 - restart service service[erchef]
192.241.179.65
192.241.179.65
192.241.179.65 Chef Client finished, 244 resources updated
192.241.179.65 chef-server Reconfigured!
192.241.179.65 Server reconfigured
192.241.179.65 -----> Bootstrapping Chef Server on cauldron.async.io is
complete.
If you want the web UI enabled, include --web-ui-enable
Wednesday, August 14, 2013
20. git server
• Several (sane) choices for hosting your own git:
• Gitolite: http://gitolite.com/gitolite/
• Gitlab: http://gitlab.org/
• Gitblit: https://code.google.com/p/gitblit/
• gitweb: https://git.wiki.kernel.org/index.php/Gitweb (please don’t)
• On windows or OSX, hosted github ($$)
• We are going to use gitolite
Wednesday, August 14, 2013
21. Gitolite
• Gitlab is cool, tries to reproduce as much of github other web-based
git hosting as possible. Installation? Several pages of hand-invoking
• Gitblit: same goal as gitlab, but built in java. So: easy install, but big and
not using the official git binaries so compatibility?
• Gitosis is dead if you run across it
• Gitweb: just don’t
Wednesday, August 14, 2013
22. Install
• Put the id_rsa.pub key of the user you want to admin gitolite as on the
cauldron server, but name it username.pub where username is the
username you want to be known as on the gitolite server
# get the software
git clone git://github.com/sitaramc/gitolite
# install it
gitolite/install -ln
# setup the initial repos with your key
gitolite setup -pk your-name.pub
Wednesday, August 14, 2013
23. Configure gitolite
• Back on your workstation machine:
git clone git@host:gitolite-admin.git
• Add your git repository setups in the cloned conf/gitolite.conf file
• Example, jonathan is the owner of the heatNode repository (and there
is a jonathan.pub RSA public key):
repo heatNode
RW+ = jonathan
Wednesday, August 14, 2013
24. You have a cauldron
• You can git add remote your cauldron plus any other git servers such as
github to any git repository you have
git remote add origin git@cauldron.your.domain:name_of_your_repo.git
git remote add github git@github.com:name_of_your_repo.git
• The name after git remote add is arbitrary! “origin” is a convention but not
required
• You git push/pull from all external servers so you have redundant copies
• Your cauldron is now a chef server
Wednesday, August 14, 2013
25. Put a real cert on the box
• As of omnibus Chef 11, webserver is nginx. Edit the ssl config to put a
real server on to get rid of the big red untrusted cert warning
• Check out http://chr4.org/blog/2013/08/01/howto-use-chef-with-ssl/
for steps on how to do it with chef
• Probably a good idea, as chef-ctl-reconfigure might blast manual
changes
Wednesday, August 14, 2013
26. Thank you. Questions?
Also, thanks to @nathenharvey and @devopsdc for letting me present, @devopsdc
and @fnichol (Fletcher Nichol) for the awesome real-time interactive improvements
to my presentation, and @fnichol for the awesome knife-server plugin!
Wednesday, August 14, 2013