Develop Commit Build Stage Deploy Towards Continuous Deployment with Django

Towards Continuous Deployment with Django Roger Barnes @mindsocket roger@mindsocket.com.au PyCon Australia 2012 1

Beer Adventure Photography Frisbee Web development Python/DjangoCo-founder @ Arribaa BTech ICS 1997 1999 2001 2003 2005 2007 2009 2011

Concepts from other talks● The why – Lean startups and customer discovery – Do more with less● The how – EC2 / provisioning – Architecture – Developing for cloud deployment – Monitoring live web applications – IaaS vs PaaS

What is continuous delivery*"Rapid, incremental, low-risk delivery of high quality, valuablenew functionality to users through automation of the build,testing and deployment process" - Jez Humble Deploying every good version of your software... … or at least being able to "Minimising MTTBID (Mean Time to Bad Idea Detection)" – Etsy * delivery == deployment for the purposes of this talk

More Than Technology● Technology - For automation● People - Cross-functional team, organised around product● Processes – Conventions and some glue that technology cant automate

Why● Fail fast, win fast - Build, Measure, Learn● Competitive advantage● Avoid YAGNI - do less● Less manual process, less risk● Real-time control – Deploy on demand – Decoupled deployment and release – Self service environments

Continuous Delivery in Practice● Single path to production● Optimise for resilience● Get comfortable with being uncomfortable● Automate as much as possible● If it hurts, do it more often● Becomes natural, return on investment is fast/high● Lots of ways to do it, heres what Ive done...

Develop Commit Build Stage Deploy Measure Environments● Make dev/test/prod as similar as possible – Full stack, no "./manage.py runserver" – Some exceptions make sense. eg for development: ● dummy email backend ● dummy message broker ● fewer threads/workers ● less memory ● less analytics instrumentation

Develop Commit Build Stage Deploy Measure Environments Dev/test virtualised using VagrantCreate and configure lightweight, reproducible, and portable development environments $ vagrant up $ vagrant ssh

Develop Commit Build Stage Deploy Measure EnvironmentsVagrantfile configures base image, provisioning, shared folders, forwarded ports Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/vagrant-manifests" puppet.manifest_file = "dev.pp" puppet.module_path = "puppet/modules" end config.vm.forward_port 80, 8000 config.vm.forward_port 3306, 3306 config.vm.share_folder "arribaa", "/opt/django-projects/arribaa", ".." end

Develop Commit Build Stage Deploy Measure Environments● Repeatable, versioned configuration – Snowflake bad, phoenix good – Puppet (masterless) for provisioning OS and services – Fabric for scripted tasks, eg: ● copy database from production ● update requirements ● migrate database ● commit and push to repository● Anti-pattern – Cant spin up new environment with one command

Develop Commit Build Stage Deploy Measure Development top level puppet configinclude uwsgiinclude statsdinclude solrinclude memcachedinclude rabbitmq Test configuration is the sameinclude nginxinclude testing Production adds:include arribaa backupinclude arribaa::db_node monitoringinclude arribaa::nginxinclude arribaa::celery

Develop Commit Build Stage Deploy Measure Using Fabric with Vagrantdef vagrant(): # get vagrant ssh setup vagrant_config = _get_vagrant_config() env.key_filename = vagrant_config[IdentityFile] env.hosts = [%s:%s % (vagrant_config[HostName], vagrant_config[Port])] env.user = vagrant_config[User]def _get_vagrant_config(): with lcd(../vagrant): result = local(vagrant ssh-config, capture=True) conf = {} for line in iter(result.splitlines()): parts = line.split() conf[parts[0]] = .join(parts[1:]) return conf Based on https://gist.github.com/1099132

Develop Commit Build Stage Deploy Measure Dependencies● virtualenv – separate env for each app● pip – install dependencies into virtualenv – use requirements file – use explicit versions ● for repository based dependencies, use commit id ● or fork on github

Develop Commit Build Stage Deploy Measure Dependencies● Using virtualenv and pip with fabricenv.site_dir = /opt/django-projects/arribaaenv.app_dir = /opt/django-projects/arribaa/arribaaenv.pip_file = requirements.txtdef ve_run(command, func=run, base_dir=env.app_dir, *args, **kwargs): with cd(base_dir): with prefix("source /opt/virtualenvs/%s/bin/activate" % env.virtualenv): return func(command, *args, **kwargs)def update_reqs(): ve_run(pip install -r %s % env.pip_file, base_dir=env.site_dir)

Develop Commit Build Stage Deploy Measure Database migration● South – Schema changes – Data migrations● Deploy separate expand and contract operations, eg: – Expand ● add new column (update model, manage.py schemamigration …) ● map from old column (manage.py datamigration …) – Deploy – Contract (optional) ● remove old column ( update model, manage.py schemamigration …)

Develop Commit Build Stage Deploy Measure Database migration● Using South with fabricdef syncdb(): ve_run("python manage.py syncdb --noinput --migrate")

Develop Commit Build Stage Deploy Measure Tying it all together● Update pip requirements on vagrant VM $ fab vagrant update_reqs● Run data migration on vagrant VM $ fab vagrant syncdb

Develop Commit Build Stage Deploy Measure Source control● Pick one, learn it● Use tags to keep track of build status● Avoid long-lived branches – or integrate them – more overhead ✔ application code● Tracks all code/documentation ✔ deployment code ✔ provisioning code ✔ configuration code ✔ documentation ✗ not build artifacts

arribaa Application code├── apps Django apps│ └── ...├── static Django static files│ └── ...├── templates Django templates│ └── ...├── fabfile.py Fabric scripts├── requirements.txt Pip└── ...bin└── jenkins.sh Jenkins jobdocs└── ...

vagrant├── Vagrantfile Dev VM config└── puppet ├── modules Puppet modules │ ├── arribaa │ ├── backup │ ├── graphite │ ├── ... │ ├── uwsgi │ └── wget └── vagrant-manifests ├── dev.pp Puppet dev ├── prod.pp Puppet prod* └── test.pp Puppet test * Not used by vagrant, but convenient to store here

Develop Commit Build Stage Deploy Measure Automated Testing● Django’s test framework● Continuous Integration and Testing – Jenkins● django-jenkins – tests – pylint – coverage – css/jslint● factory_boy instead of fixtures

Develop Commit Build Stage Deploy Measure Test separation● Split up tests by type● Can parallelise, keeping test run times low – < 20 minutes from commit to production● Unit tests first, faster feedback● Run tests on different triggers, keep some out of the main build● Etsy - Divide and Concur – http://codeascraft.etsy.com/2011/04/20/divide-and-concur/

Develop Commit Build Stage Deploy Measure Test separation● Unit● Functional/UI – Selenium et al● Staging and production smoke tests – Crawl URLs – Load/performance testing● Flaky tests, Slow tests – exclude from regular build, run on a separate schedule

Develop Commit Build Stage Deploy Measure Test separation ● How to do this with Django? – Suites – Custom test runner – Other test systems (Nose has an attrib plugin) ● My current solution – Annotating tests, and adapted django-jenkins commandclass TestBooking(TestCase): @pipelineTag ("unit") def test_when_lapsed(self): oldbooking = factories.BookingFactory.build(when=some_old_date) self.assertTrue(oldbooking.when_lapsed())

Develop Commit Build Stage Deploy Measure Test separation# Used to ignore testsignore = lambda test_item: Nonedef pipelineTag(*args): """ Let a testcase run if any of the tags are in sys.argv and none of the tags match not-* in sys.argv. """ # Abstain if were not in a build pipeline (ie using django-jenkins) # or if no tag(s) specified if jenkins_pipeline not in sys.argv or not any([tagged- in arg for arg in sys.argv]): return _id tags = args assert set(tags) < set([unit, functional, flaky, slow, staging]) #Silently ignore if no matching tag if not any([tagged-+tag in sys.argv for tag in tags]): return ignore # Skip if "not-blah" tagged if any([not-+tag in sys.argv for tag in tags]): return skip(tagged: + ,.join([tag for tag in tags if not-+tag in sys.argv])) # This test made it through return _id

Develop Commit Build Stage Deploy Measure Build Pipeline● One pipeline● All developers feed start of pipeline via VCS (master) – Improve policy of "dont deploy broken code"... – ...with system where its harder to deploy broken code● Jenkins Build Pipeline Plugin – Series of dependant jobs, each runs different tests – Feeds into pre-deploy staging job – Successful staging job enables deploy job

Develop Commit Build Stage Deploy Measure Build Pipeline Currently simple linear flow

Develop Commit Build Stage Deploy Measure Build Pipeline● Jenkins job – unit test stage: bash -ex ./bin/jenkins.sh tagged-unit not-flaky not-slow● jenkins.sh # Setup virtualenv, pip requirements, settings.py etc ...snip... cd $WORKSPACE/arribaa python manage.py clean_pyc python manage.py jenkins_pipeline "$@"

Develop Commit Build Stage Deploy Measure Staging/QA Job● fab staging-deploy – very similar process to production deploy● Test run of production deployment using Fabric – Deploy code and config to staging environment (Vagrant VM) – Apply dependency updates ● puppet config ● pip requirements – Copy database from production and run data migration● Should now have a deployed QA environment● Browser smoke tests – currently selenium – Not using Django 1.4 LiveTestCase here, we have a full stack to play with● Tag successful build as deployable

Develop Commit Build Stage Deploy Measure Deployment Strategies● Optimistic – Deploy, reload services – some appservers handle this well● Blue/Green – Parallel environments, 1 idle – Idle environment gets new version – Smoke tests and monitoring before switching over● Canary – Take subset of app servers out of pool – Deploy to subset – Smoke tests and monitoring before deploying rest of cluster● Deploy whole new (app) server instance – then rewire load-balancer

Develop Commit Build Stage Deploy Measure Deployment with Jenkins & FabricUsing Git tags – Set in Jenkins after successful staging run – Used by deploy in Fabricdef pull(): with cd(env.site_dir): run(git fetch) latest_tag = run(git describe --tags --match "ci-passed-staging-*" origin/master, pty=False) run(git reset --hard refs/tags/%s % latest_tag, pty=False)

Develop Commit Build Stage Deploy Measure Tags set by build pipeline

Develop Commit Build Stage Deploy Measure Deployment with Jenkins & FabricJenkins script: virtualenv -q /opt/virtualenvs/arribaa-jenkins source /opt/virtualenvs/arribaa-jenkins/bin/activate pip install -r requirements.txt pip install -r requirements-testing.txt cd $WORKSPACE/arribaa fab deploy

Develop Commit Build Stage Deploy Measure Deployment with Jenkins & Fabricdef deploy(): dbbackup() # django-dbbackup – to dropbox changes = pull() # git fetch, git describe, git log, git reset apply_puppet() # puppet apply … prod.pp update_requirements() # pip install -r requirements.txt clean_pyc() # django-extensions command syncdb() # and –migrate (South) collectstatic() reload_uwsgi() # reload app server restart_celeryd() # bounce task server run(git tag -f ci-deployed && git push –tags) # tag deployed version send_email(changes) # send notification

Develop Commit Build Stage Deploy Measure Decoupling deployment and release● Feature flags vs feature branches – Real time control with flags – Cleanup/maint needed for flags – Harder/slower to test different combinations in either case● Good for testing ideas (on/off, A/B, power users, QA etc)● Gargoyle – includes optional admin interface – selective criteria (%age of users, staff only, ...)● Alternative: Django Waffle

Develop Commit Build Stage Deploy Measure Measurement● Eyes wide open – Track system and user behaviour● Post-deployment – Error, performance and behaviour heuristics – Trend monitoring – Goal conversion rates

Develop Commit Build Stage Deploy Measure Measurement

Develop Commit Build Stage Deploy Measure Measurement - Munin

Develop Commit Build Stage Deploy Measure

Develop Commit Build Stage Deploy Measure Measurement● django-statsd, statsd and graphite – page timing – counters http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

Rollback options● Level of automation – Depends on risk aversion and complexity of environment – 12+ years of deployment, average 5 deployments per week, very very few actual rollbacks● Engineer around it – lots of small changes → more likely to "roll-forward" – staged deployment → minimise harm● Engineer for it – avoid backwards incompatible data migrations – tag deployments, keep previous versions handy

Rollback options "My deployment rollback strategy is like a car with no reverse gear. You can still go backwards, but you have to get out and push.Knowing that makes you drive carefully." - Me

Where to from here Have an existing app with some gaps? Think about whats slowing you down the most● Suggested priorities – Automated tests (Django built-in) – Continuous integration (jenkins, django-jenkins) – Measure all the things (sentry, statsd + graphite, new relic) – Scripted database (South) – Scripted deployment (fabric) – Configuration management (puppet) – Virtualised development (Vagrant) – Test separation (test decorator, customise test runner) – Feature flags (gargoyle)

Looking Forward● Better version pinning● Fully automated provisioning● Better test separation and parallelisation● Combined coverage results● Reduce dependence on external services● Make deployment more atomic● Staged rollouts to cluster – canary deployment● Make the "big red deploy button" big and red

Resources● Slides – http://slideshare.net/mindsocket/● Continuous delivery/deployment – Continuous Delivery Primer - http://www.informit.com/articles/article.aspx?p=1641923 – Anatomy of the Deployment Pipeline - http://www.informit.com/articles/printerfriendly.aspx?p=1621865 – Code as Craft – Etsy http://codeascraft.etsy.com/ – Safe deploying on the cutting edge – Urban Airship http://lanyrd.com/2011/djangocon-us/shbqz/● Vagrant – http://vagrantup.com● Vagrant with Fabric - https://gist.github.com/1099132● Jenkins Build Pipeline Plugin - http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html● Git/fabric based rollback options – http://dan.bravender.us/2012/5/11/git-based_fabric_deploys_are_awesome.html – http://www.askthepony.com/blog/2011/07/setup-a-complete-django-server-deploy-rollback-%E2%80%93-all-in-one- powerful-script/ – http://lethain.com/deploying-django-with-fabric/

Thank You!http://slideshare.net/mindsocket Questions?