This document discusses Docker in production at DramaFever. Some key points: - Docker provides consistent development and repeatable deployments. - DramaFever uses a private Docker registry hosted on S3 for image storage. They faced scaling issues with a central registry server. - Jenkins is used to build Docker images from source code and push them to the private registry for deployment. - Docker is used throughout the development and deployment process, from local development to production. It has provided benefits like faster builds and easier configuration management.

1. @bridgetkromhout
Docker in Production
Reality, Not Hype
Bridget Kromhout

2. @bridgetkromhout
Bridget Kromhout
Operations Engineer
@DramaFever
Minneapolis, Minnesota
@devopsdays
@devopsdaysMSP
@arresteddevops
bridgetkromhout.com

3. @bridgetkromhout
K-dramas since 2009. Docker in prod since October 2013.

4. @bridgetkromhout
Why Docker?
consistent development repeatable deployment

5. How?
not:
a tutorial
but:
repeatable
@bridgetkromhout

6. private registry:
the official party line
https://github.com/docker/docker-
registry#quick-start
S3 is a storage engine option
but… a central registry server didn’t scale well
for us
@bridgetkromhout

7. private registry: dramafever
@bridgetkromhout
# this goes in /etc/default/docker to control
docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-
registry=local-repo-alias.com:5000"
● local-repo-alias.com in DNS with A record to 127.0.0.1
● OS X /etc/hosts: use the boot2docker host-only network IP

8. registry upstart
docker pull public_registry_image
docker run -p 5000:5000 --name registry
-v /etc/docker-reg:/registry-conf
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml
public_registry_image
@bridgetkromhout

9. config.yml
s3_region: us-east-1
s3_access_key: <aws-accesskey>
s3_secret_key: <aws-secretkey>
s3_bucket: <bucketname>
standalone: true
@bridgetkromhout

10. what even is flate?!
Pulling repository local-repo-alias.com:5000/www
4dda2b433370: Error pulling image (prod) from
local-repo-alias.com:5000/www, flate: corrupt
input before offset 54393671 flate: corrupt
input before offset 54393671
d497ad3926c8: Error downloading dependent layers
2014/12/07 02:34:54 Error pulling image (prod)
from local-repo-alias.com:5000/www, flate:
corrupt input before offset 54393671
@bridgetkromhout

11. registry rewrite coming!
DOCKER_OPTS="--graph=/mnt/docker --
insecure-registry=local-repo-alias.
com:5000 -e STORAGE_REDIRECT=true"
...until we get to the promised go lan(d|g), there’s a
workaround for the flate errors we’re seeing:
@bridgetkromhout

12. Achievement
unlocked:
distributed
private
Docker
registry
@bridgetkromhout

13. @bridgetkromhout
Next up:
build pipeline
starring
everyone’s
favorite butler

14. weekly base builds
FROM local-repo-alias.com:5000/www-base
● include infrequently-changing
dependencies
○ ubuntu packages
○ pip requirements
○ wheels
● other builds can start from these images
(so they’re faster):
@bridgetkromhout

15. sudo docker build -t="a12fbdc" .
sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 --
name test.a12fbdc a12fbdc py.test -s
sudo docker tag a12fbdc local-repo-alias.com:
5000/www:'dev'
sudo docker push local-repo-alias.com:5000/www:'dev'
@bridgetkromhout
www-master build

16. 2014/10/30 21:35:31 Error getting container init rootfs
b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f
3108f86386aab67 from driver devicemapper: open
/dev/mapper/docker-9:127-14024705-
b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f
3108f86386aab67-init: no such file or directory
make: *** [build-django] Error 1
Build step 'Execute shell' marked build as failure
@bridgetkromhout
breaking builds

17. https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
@bridgetkromhout

18. @bridgetkromhout
Retry the build…
...only if a specific regex appears

19. @bridgetkromhout
useful for unattended
base builds
need to change how it
reports to Slack

20. @bridgetkromhout
tag for staging
tag for prod
out of ELB
restart upstart
back in ELB
Ship it!

21. What
about
local
development?
@bridgetkromhout

22. before summer 2014
Vagrant for local development
chef-solo provisioner
17 minutes to install everything
@bridgetkromhout

23. now: boot2docker
devs pull down images built on jenkins
mysql image is built with fixtures
can run master or qa image (or even prod)
can build new local images from Dockerfiles
@bridgetkromhout

24. local registry for dev
docker run -d -p 5000:5000 --name
docker-reg -v ${DFHOME}:${DFHOME} -e
DOCKER_REGISTRY_CONFIG=${DFHOME}
/config/docker-registry/config.yml
public_registry_image
@bridgetkromhout

25. $ boot2docker ssh date -u
Mon Nov 24 16:09:02 UTC 2014
$ date -u
Tue Nov 25 01:43:49 UTC 2014
@bridgetkromhout
time is what turns kittens into cats

26. S3 requires clock sync
$ docker pull local-repo-alias.com:5000/mysql
Pulling repository local-repo-alias.com:5000/mysql
2014/11/24 19:44:31 HTTP code: 500
$ boot2docker ssh sudo date --set "$(env
TZ=UTC date '+%F %H:%M:%S')"
@bridgetkromhout

27. Devs can use their preferred editing environment:
-v ${DFHOME}/www:/var/www
We still want logs, too, so we expose those for the dev here:
-v ${DFHOME}/www/run:/var/log
volume mounting & our fork
@bridgetkromhout
Until 1.3 we ran a forked boot2docker
We needed to mount local files into the VM

28. containerizing front-end
useful for building front-end apps on Jenkins
also allows consistent testing
RUN apt-get install -y nodejs nodejs-legacy npm
RUN npm install -g grunt-cli@0.1.13
RUN npm install -g bower@1.3.8
RUN npm install -g phantomjs@1.9.7-14
ADD bower.json /var/www/dependencies/bower.json
RUN cd /var/www/dependencies && bower install --
allow-root
--config.interactive=false --force
@bridgetkromhout

29. @bridgetkromhout
django:
image: local-repo-alias.com:5000/www:dev
ports:
- "8000:8000"
links:
- mysql
- redis
environment:
- PYTHONPATH=/var/local
- DJANGO_ENVIRON=LOCAL
- DB_PORT_3306_TCP_ADDR=mysql
command: /var/local/config/local/start-django-local
volumes:
- ${DFHOME}/www/run:/var/log
- ${DFHOME}/www:/var/local
mysql:
image: local-repo-alias.com:5000/mysql:dev
expose:
- "3306:3306"

30. for persistent instances
# remove stopped containers
@daily docker rm `docker ps -aq`
# remove images tagged "none"
@daily docker rmi `sudo docker images | grep none
| awk -F' +' '{print $3}'`
@bridgetkromhout

31. failure modes
cron zombies
out of memory errors
race conditions
@bridgetkromhout

32. what isolation?
-v /var/log/containers:/var/log
@bridgetkromhout
Host instances moving into

33. through a container darkly: monitoring
@bridgetkromhout

34. containers building
(lighter) containers
easier with
statically linked
binaries
go microservices
android apk
@bridgetkromhout

35. $ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB
local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB
local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB
google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB
public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB
scratch latest 511136ea3c5a 18 months ago 0 B
$
ever-smaller images
@bridgetkromhout

36. @bridgetkromhout
www.dramafever.com/company/careers.html
Thank you!
(and we’re hiring!)