DevOps is the new rage among system administrators, applying agile software development techniques to infrastructure configuration management. In the center of the DevOps movement is the open-source Chef tool, implemented in Ruby atop CouchDB. Unsatisfied with the performance of the open-source and/or hosted Chef server and needing better integration with our Python web application, we set out to build a new implementation in Python atop MongoDB. This talk will give you an overview of Chef, reasons for doing a new implementation, and lots of code examples of how we made it all work together to get a chef server that screams.

1. Rick
Copeland
@rick446
Arborian
Consulting,
LLC

2.  Infrastructure
as
Code
 Resources
&
Providers
 Cookbooks,
Recipes,
Clients,
and
Nodes

3. CouchDB
Solr
Solr
Indexer
Chef
Server
RabbitMQ
RabbitMQ
RabbitMQ
(API)
HTTP
REST
API
Chef
Server
knife
chef-­‐client
(Web
UI)

4. Build,
register,
and
authenticate
the
node
Synchronize
cookbooks
Build
resource
collection
(run
the
recipes
in
order)
Configure
node
(“converge”)
Run
notification
handlers

5. CouchDB
Solr
Ruby
Solr
Indexer
Chef
Server
RabbitMQ
RabbitMQ
RabbitMQ
(API)
Ruby
Ruby
HTTP
REST
API
Chef
Server
knife
chef-­‐client
(Web
UI)

6.  Chef
assumes
a
bootstrapped
node
exists
 Chef
doesn’t
keep
release
notes
 Code
and
infrastructure
are
versioned
differently
 Solution:
Web
app
to
manage
deployments
&
generate
release
notes

7. MonQ
MongoDB
Solr
Indexer
Python
Chef
Server
Solr
(API
+
web)
Ruby
HTTP
REST
API
knife
chef-­‐client

8.  Reduce
#
of
processes
&
technologies
 Don’t
know
Ruby
well
 Keep
private
keys
out
of
the
system
 Integrate
with
existing
authentication
 Performance

9. /clients
/nodes
/data
/environments
/roles
/sandboxes
/cookbooks
/search

10.  Mostly
JSON

almost
BSON
 References
to
Ruby
files
stored
separately
{
"name":
"allura-­‐0.0.1”,…
"json_class":
"Chef::CookbookVersion",
Ruby
files
stored
on
S3
"attributes":
[
{
"name":
"default.rb",
"url":
"https://s3.amazonaws.com/opscode-­‐platform-­‐production-­‐data/…
"checksum":
"749a3a328d6c47a32d134b336183981f",
"path":
"attributes/default.rb",
"specificity":
"default”…

11. MongoDB
Validator
role = collection(
'chef.role', doc_session,
Field('_id', S.ObjectId()),
Field('account_id', S.ObjectId(if_missing=None)),
Shorthand
with
Field('name', str),
Python
Types
Field('description', str),
Field('default_attributes', str),
Field('override_attributes', str),
Field('run_list', [ str ] ),
Field('env_run_lists', { str: [ str ]}),
Index('account_id', 'name', unique=True))
Embedded
Index
Definitions
Documents

12. Models
know
where
they
live
class Role(object):
def url(self):
return request.relative_url(
config.chef_api_root + '/roles/' + self.name)
Models
can
be
def __json__(self):
return dict(
turned
into
dict
(to
chef_type='role', be
JSONified)
json_class='Chef::Role',!
…
default_attributes=loads(self.default_attributes),!
…)
Models
can
be
def update(self, d):
self.name = d['name'] updated
from
dict
…
self.default_attributes = dumps(d['default_attributes'])
self.override_attributes = dumps(d['override_attributes'])!
…

13. class RoleSchema(JSONModelSchema):!
model_class=CM.role!
chef_type='role’!
json_class='Chef::Role’!
exclude_fields=['_id', 'account_id']

14.  /foo/bar/baz

root.foo.bar.baz
 Lots
of
decorators
 Validation
(params/body

**kwargs)
 Authorization
 Rendering
 HTTP
method
lookup
 GET
/foo/bar

root.foo.bar._get()

15. Returns
JSON
class RolesController(RESTController):
@expose(template_engine='json')
def _get(self):…
Only
admins
can
access
@expose(template_engine='json',
acl=[CACE.admin(True), ACE.any(False)],
schema=cv.RoleSchema)
Convert
and
def _post(self, **kwargs):… Validate
POST
def __getattr__(self, name):
return RoleController(name) Continue
dotted
lookup

16. class RoleController(RESTController):
@expose(template_engine='json')
def _get(self):
… PUT
looks
just
like
a
POST
@expose(template_engine='json',
acl=[CACE.admin(True), ACE.any(False)],
schema=cv.RoleSchema)
def _put(self, name, **kwargs):
…
@expose(template_engine='json',
acl=[CACE.admin(True), ACE.any(False)])
def _delete(self):
…

17. class RoleController(RESTController):!
AttributeError

def __init__(self, name):
HTTP
404
Not
Found
self._role = CM.Role.query.get(
account_id=c.account._id,
name=name)
if self._role is None:
raise AttributeError, name!
@expose(template_engine='json')
def _get(self):
return self._role!
Auto-­‐JSONify

18. class RoleController(RESTController):!
…
@expose(template_engine='json',
acl=[CACE.admin(True), ACE.any(False)],
Update
model
schema=cv.RoleSchema) from
kwargs
def _put(self, name, **kwargs):
assert name == self._role.name
self._role.update(kwargs)
return self._role!
@expose(template_engine='json',
acl=[CACE.admin(True), ACE.any(False)])
def _delete(self):
self._role.delete()
return self._role

19.  Don’t
trust
the
docs
 Don’t
trust
the
docs
▪ Don’t
trust
the
docs
 Use
fat
models
 Framework
support
for
REST
&
JSON
 You’re
gonna
have
to
learn
some
Ruby
anyway
 JSON
!=
BSON

20.  Better
test
coverage
 Search
support
(SOLR
/
ElasticSearch)
 More
testing
with
real-­‐world
deployments
 Finalize
integration
with
deployment
manager

21. Rick
Copeland
@rick446
Arborian
Consulting,
LLC

22.  http://openmymind.net/2011/10/28/
CouchDB-­‐And-­‐MongoDB-­‐Performance/
 MongoDB
is
14x
faster
 http://www.snailinaturtleneck.com/blog/
2009/06/29/couchdb-­‐vs-­‐mongodb-­‐
benchmark/