Cooking Up Drama

Cooking Up Drama
Bridget Kromhout & Peter Shannon

@bridgetkromhout @pietroshannon
Bridget Kromhout Peter Shannon
@devopsdaysMSP
@arresteddevops
devopsdays.org
Go
Data
Ops & Zip Ties

@pietroshannon@bridgetkromhout
DramaFever: streaming international content

peak load: tens of thousands of requests per second
traffic variance: swings 10-20x throughout the week

Architecture
● All services in AWS
● Python (Django) main website
● Go microservices (video analytics ingest, on-
the-fly image processing, bookmarking…)
● Upstreams routed via nginx
● Celery + SQS for async tasks
● Streaming delivery through Akamai

What do you mean, NO CM!?
Docker
Jenkins
Graphite & ELK
AWS
Config Management

CONFIG MANAGEMENT
EVERYWHERE

Why did we need CM?
● Single source of truth to build AMIs and
provision AWS instances.
● Consistent configuration across ephemeral
instances.
● Hand-crafted, longer-lived instances are
hard to reproduce.

CM selection showdown: fight!

cooperation, not competition

decisions: hosted chef

decisions: hosted chef
(but not “Starter Kit”)

decisions: environments

roles

roles
minimal notifies

workflow

workflow: chef-dk

workflow: chef-dk
test kitchen

workflow: chef-dk
test kitchen
jenkins

workflow: chef-dk
test kitchen
jenkins
via
github

workflow: chef-dk
test kitchen
jenkins
via
github
rubocop
foodcritic

Docker images
built from Dockerfiles
in Jenkins jobs
deployment
via fabric & upstart

pain points?
where to start...

pain points?
visible wins?
where to start...

where to start...
production!

autoscaling: scorched earth

#!/bin/bash
cat <<EOF > /etc/init/django.conf
description "Run Django containers for www"
start on started docker-reg
stop on runlevel [!2345] or stopped docker
respawn limit 5 30
[...]
replacing 100s of lines of userdata...

#!/bin/bash
# upstart configs are now created by chef
rm /etc/chef/client.pem
mkdir -p /var/log/chef
chef-client -r 'role[rolename]' -E 'environment' -L
/var/log/chef/chef-client.log
...with a chef-client run.

deregistering nodes (knife)
/usr/bin/knife node delete -y -c /etc/chef/knife.rb
<%= node['base']['chefname'] %>
/usr/bin/knife client delete -y -c /etc/chef/knife.rb
<%= node['base']['chefname'] %>

deregistering nodes (rc script)
template '/etc/init.d/unregister_chef_instance' do
source 'default/unregister_chef_instance.erb'
end
link '/etc/rc0.d/K99unregister_chef_instance' do
to '/etc/init.d/unregister_chef_instance'
end

ami factory
Autoscaling
Packer
AMI
EC2 Instances
Jenkins
GitHub
Chef

apt-get install:
only for AMI...

apt-get install:
only for AMI…
...and test kitchen

Docker: what to chef?
not Docker images...
not application settings...
LWRP:
upstarts
admin wrappers

Docker &
LWRPs
docker run
-e DJANGO_ENVIRON=PROD
-e HAPROXY=df/haproxy-prod.cfg
-p 8000:8000
-v /var/log/containers:/var/log
--name django
localhost-alias.com:5000/www:prod
/var/www/bin/start-django

docker run
<% if @docker_rm == true -%>
--rm
<% end %>
<% @docker_env.each do |k, v| -%>
-e <%= k %>=<%= v %>
<% end %>
<% @docker_port.each do |p| -%>
-p <%= p %>
<% end %>
upstart
template

<% @docker_volume.each do |v| -%>
-v <%= v %>
<% end %>
--name <%= @application_name %>
localhost-alias.com:<%= @registry_port %>/<%=
@docker_image %>:<%= @docker_tag %>
<%= @docker_command %>
upstart
template
(cont)

attribute :docker_command, :kind_of => String, :required => true
attribute :docker_env, :kind_of => Hash, :default => {}
attribute :docker_port, :kind_of => Array, :default => []
attribute :docker_volume, :kind_of => Array, :default =>
['/var/log/containers:/var/log']
attribute :docker_rm, :kind_of => [TrueClass, FalseClass], :
default => false
attribute :docker_name_override, :kind_of => String, :default =>
nil
attribute :docker_image_override, :kind_of => String, :default
=> nil
attribute :docker_tag_override, :kind_of => String, :default =>
nil
so many overrides

if new_resource.docker_name_override.nil?
name = new_resource.name
upstart = name
else
name = new_resource.docker_name_override
upstart = "#{new_resource.name}-#{name}"
end
if this, then just don’t

attribute :command, :kind_of => String, :required => true
attribute :env, :kind_of => Hash, :default => {}
attribute :port, :kind_of => Array, :default => []
attribute :volume, :kind_of => Array, :default =>
['/var/log/containers:/var/log']
attribute :rm, :kind_of => [TrueClass, FalseClass], :default
=> false
attribute :image, :kind_of => String, :required => true
attribute :tag, :kind_of => String, :required => true
attribute :type, :kind_of => String, :required => true
attribute :cron, :kind_of => [TrueClass, FalseClass], :default
=> false
using attributes

recipe using LWRP
base_docker node['www']['django']['name'] do
command node['www']['django']['command']
env node['www'][service]['django'][env]['env']
image node['www']['django']['image']
port node['www'][service]['django'][env]['port']
tag node['www'][service]['django'][env]['tag']
type node['www']['django']['type']
end

Wrapping
Community Cookbooks
cookbook_path ["#{ENV['DFHOME']}
/chef/cookbooks", "#{ENV['DFHOME']}
/chef/community_cookbooks"]
clear delineation
no forking of community cookbooks

wrapper cookbook - leroy
● depends on community cookbooks (jenkins,
etc)
● recipes include: builds, packer, plugins
● fun with FC001
● variations on our base (fstab, docker registry)

private docker registry
# this goes in /etc/default/docker to control
docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-
registry=localhost-alias.com:5000"
● localhost-alias.com in DNS with A record to 127.0.0.1
● OS X /etc/hosts: use the boot2docker host-only network IP

registry upstart
docker pull public_registry_image
docker run -p 5000:5000 --name registry
-v /etc/docker-reg:/registry-conf
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml
public_registry_image

config.yml
s3_region: us-east-1
s3_access_key: <aws-accesskey>
s3_secret_key: <aws-secretkey>
s3_bucket: <bucketname>
standalone: true
storage: s3
storage_path: /registry

packer$HOME/packer/packer build
-var "account_id=$AWS_ACCOUNT_ID"
-var "aws_access_key_id=$AWS_ACCESS_KEY_ID"
-var "aws_secret_key=$AWS_SECRET_ACCESS_KEY"
-var "x509_cert_path=$AWS_X509_CERT_PATH"
-var "x509_key_path=$AWS_X509_KEY_PATH"
-var "s3_bucket=bucketname"
-var "ami_name=$AMI_NAME"
-var "source_ami=$SOURCE_AMI"
-var "chef_validation=$CHEF_VAL"
-var "chef_client=$HOME/packer/client.rb"
-only=amazon-instance
$HOME/packer/prod.json

packer for ami building
{
"type": "chef-client",
"server_url": "https://api.opscode.
com/organizations/dramafever",
"run_list": [ "base::ami" ],
"validation_key_path": "{{user `chef_validation`}}",
"validation_client_name": "dramafever-validator",
"node_name": "packer-ami"
}

limiting packer IAM permissions
"Action":[
"ec2:TerminateInstances",
"ec2:StopInstances",
"ec2:DeleteSnapshot",
"ec2:DetachVolume",
"ec2:DeleteVolume",
"ec2:ModifyImageAttribute"
],
"Effect":"Allow",
"Resource":"*",
"Condition":{
"StringEquals":{
"ec2:ResourceTag/name":"Packer
Builder"
}
}

so, about that Jenkins server...

(un-)containerize some of the things

sentrybase_docker_upstart 'sentry-udp' do
docker_env(
'DJANGO_ENVIRON' => 'DRAMAFEVER'
)
docker_image_override 'sentry'
docker_tag_override 'dev'
docker_port ['9001:9001/udp']
docker_command 'sentry --config=/config/sentry.conf.py start
udp'
end

chef for devs

self-service visibility

(re)cycle of life

thanks!

Cooking Up Drama

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (13)

Ähnlich wie Cooking Up Drama

Ähnlich wie Cooking Up Drama (20)

Mehr von bridgetkromhout

Mehr von bridgetkromhout (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Cooking Up Drama