Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Safely         Deploying on         the cutting edge         Eric Holscher         Urban Airship         Djangocon 2011Wed...
Wednesday, September 7, 2011
Talk Contents         •   Company culture & process         •   Deployment environment         •   Tools for deploying    ...
ProcessWednesday, September 7, 2011
Process         •   Deploy out of git         •   Standard git-based production/master branch model         •   Production...
Features         •   Easily allows you to hot-fix production         •   Keep a stable master         •   Run CI on the mas...
Services         •   Everything that we deploy is conceptualized as a service         •   Services all live in /mnt/servic...
QA Environment         •   Run all of your master branches         •   Allow you to get a copy of what will become product...
Deployment Design GoalsWednesday, September 7, 2011
Jump machine         •   Have a standard place for all deployments to happen         •   Log all commands runWednesday, Se...
No External Services         •   Chishop         •   No external server required to deploy code         •   All branches a...
Services look the same         •   Python         •   Java         •   “Unix”Wednesday, September 7, 2011
Composable         •   Small pieces that you can build into better things         •   Useful when trying to do something y...
EnvironmentWednesday, September 7, 2011
Environment         •   Where code lands on the remote machine         •   Mimics a chroot         •   Uses virtualenv & s...
File Structure         •   /mnt/services/airship             •   bin/             •   current -> deploy-2011-08-03_14-37-3...
etc/         •   supervisord.conf             •   [include]             •   files = *.conf         •   airship.confWednesda...
bin/         •   start         •   stop         •   restart         •   logsWednesday, September 7, 2011
SCRIPT_DIR=$(dirname $0)         SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd)         cd $SERVICE_DIR         supervisor...
Bin scripts         •   All of the process-level binscripts wrap supervisord         •   bin/start -> supervisordctl start...
var/         •   data/         •   log/         •   run/         •   tmp/Wednesday, September 7, 2011
Init.d         •   All services share a common init.d script         •   This init.d script calls into the service’s bin/ ...
SERVICE_USER=<%= @service %>         SERVICE_NAME=<%= @service %>         SERVICE_PATH=/mnt/services/$SERVICE_NAME        ...
ToolsWednesday, September 7, 2011
Tools         •   Fabric         •   Rsync         •   Pip         •   VirtualenvWednesday, September 7, 2011
Low-level verbs         •   pull         •   build         •   tag         •   sync         •   install         •   rollba...
Pull         •   Update the code from the source repository         •   Defaults to the “production” branch             • ...
Build         •   Could be called “prepare”         •   Do local-specific things to get repo into a ready state         •  ...
Tag         •   Set a tag for the deploy in the git repo         •   If the current commit already has a tag, use that ins...
Sync         •   Move the code from the local to the remote box         •   Uses rsync to put it into the remote service d...
Install         •   Make the code the active path for code on the machine         •   This is generally installing code in...
Rollback         •   When you break things, you need to undo quickly         •   Reset the repository to the previous depl...
Start/Stop/Reload         •   Allow you to bounce services as part of deployment         •   Allow reload for services tha...
CLI UI         •   Have nice wrapper commands that do common tasks         •   deploy host:web-0 full_deploy:airship      ...
UI cont.         •   deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
#!/bin/bash         cd ~/airdeploy         DATE=$(date +%Y_%-m_%-d-%H-%m-%s)         echo "deploy" $@ > logs/$DATE.log    ...
Meta-commands         •   Hard-code the correct deployment behavior         •   “Make easy things easy, and wrong things h...
MagicifyingWednesday, September 7, 2011
Magicifying         •   Now that we have a solid base, we can automate on top         •   When you do a meta deploy, it sh...
Workflow         •   Deploy to one web server, preferably with one worker         •   Restart it         •   Run it against...
Heuristics         •   Any 500s         •   Number of 200s to non-200s         •   Number of 500s to 200s         •   Requ...
How it works         •   Tell load balancer to take machine out of pool             •   /take_me_out_of_the_lb -> 200     ...
Thanks         •   Alex Kritikos         •   Erik Onnen         •   SchmichaelWednesday, September 7, 2011
Questions?         •   Eric Holscher         •   Urban Airship (Hiring and whatnot)         •   eric@ericholscher.comWedne...
Nächste SlideShare
Wird geladen in …5
×

Deploying on the cutting edge

Talk on how we do deployment at Urban Airship from Djangocon 2011.

Ähnliche Bücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen
  • Als Erste(r) kommentieren

Deploying on the cutting edge

  1. 1. Safely Deploying on the cutting edge Eric Holscher Urban Airship Djangocon 2011Wednesday, September 7, 2011
  2. 2. Wednesday, September 7, 2011
  3. 3. Talk Contents • Company culture & process • Deployment environment • Tools for deploying • Verifying deploymentWednesday, September 7, 2011
  4. 4. ProcessWednesday, September 7, 2011
  5. 5. Process • Deploy out of git • Standard git-based production/master branch model • Production branch has releases are tagged with timestamp • deploy-2011-08-03_14-37-38 • Feature branches • http://nvie.com/posts/a-successful-git-branching-model/Wednesday, September 7, 2011
  6. 6. Features • Easily allows you to hot-fix production • Keep a stable master • Run CI on the master branch or long-lived feature branchesWednesday, September 7, 2011
  7. 7. Services • Everything that we deploy is conceptualized as a service • Services all live in /mnt/services/<slug> (Thanks ec2) • A service is an instance of a repository on a machine • A repository might have multiple services • eg. Airship deployed into “celery” and “web” services • This maps really well onto Chef cookbooksWednesday, September 7, 2011
  8. 8. QA Environment • Run all of your master branches • Allow you to get a copy of what will become production • Catch errors before they are seen by customers • Spawn new ones for long-lived feature branches • `host web-0` and figure out based on IPWednesday, September 7, 2011
  9. 9. Deployment Design GoalsWednesday, September 7, 2011
  10. 10. Jump machine • Have a standard place for all deployments to happen • Log all commands runWednesday, September 7, 2011
  11. 11. No External Services • Chishop • No external server required to deploy code • All branches are checked out on an admin serverWednesday, September 7, 2011
  12. 12. Services look the same • Python • Java • “Unix”Wednesday, September 7, 2011
  13. 13. Composable • Small pieces that you can build into better things • Useful when trying to do something you didn’t plan forWednesday, September 7, 2011
  14. 14. EnvironmentWednesday, September 7, 2011
  15. 15. Environment • Where code lands on the remote machine • Mimics a chroot • Uses virtualenv & supervisord • Owned by the service-user • Managed by ChefWednesday, September 7, 2011
  16. 16. File Structure • /mnt/services/airship • bin/ • current -> deploy-2011-08-03_14-37-38 • deploy-2011-08-03_14-37-38 • etc/ • var/Wednesday, September 7, 2011
  17. 17. etc/ • supervisord.conf • [include] • files = *.conf • airship.confWednesday, September 7, 2011
  18. 18. bin/ • start • stop • restart • logsWednesday, September 7, 2011
  19. 19. SCRIPT_DIR=$(dirname $0) SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd) cd $SERVICE_DIR supervisorctl pid > /dev/null 2>&1 if [ "$?" != "0" ]; then echo "Supervisord not running, starting." supervisord else echo "Supervisord running, starting all processes." supervisorctl start all fi cd - > /dev/null 2>&1Wednesday, September 7, 2011
  20. 20. Bin scripts • All of the process-level binscripts wrap supervisord • bin/start -> supervisordctl start all • bin/start foo -> supervisorctl start foo • bin/stop -> supervisorctl stop all • bin/stop shutdown -> supervisorctl shutdownWednesday, September 7, 2011
  21. 21. var/ • data/ • log/ • run/ • tmp/Wednesday, September 7, 2011
  22. 22. Init.d • All services share a common init.d script • This init.d script calls into the service’s bin/ • /etc/init.d/airship start -> /mnt/services/airship/bin/startWednesday, September 7, 2011
  23. 23. SERVICE_USER=<%= @service %> SERVICE_NAME=<%= @service %> SERVICE_PATH=/mnt/services/$SERVICE_NAME set -e RET_CODE=0 case "$1" in start) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/start RET_CODE=$? ;; stop) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/stop RET_CODE=$? ;; restart) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/restart RET_CODE=$? ;; status) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/status RET_CODE=$? ;; *) echo "$SERVICE_NAME service usage: $0 {start|stop|restart|status}" ;; esac exit $RET_CODEWednesday, September 7, 2011
  24. 24. ToolsWednesday, September 7, 2011
  25. 25. Tools • Fabric • Rsync • Pip • VirtualenvWednesday, September 7, 2011
  26. 26. Low-level verbs • pull • build • tag • sync • install • rollback • start/stop/restart/reloadWednesday, September 7, 2011
  27. 27. Pull • Update the code from the source repository • Defaults to the “production” branch • def pull(repo=None, ref=origin/production) • Can pass in a specific revision/branch/tag/hashish • local(git reset --hard %s % ref, capture=False)Wednesday, September 7, 2011
  28. 28. Build • Could be called “prepare” • Do local-specific things to get repo into a ready state • Mostly used for compiling in java-land • Useful in Python for running pre-install tasksWednesday, September 7, 2011
  29. 29. Tag • Set a tag for the deploy in the git repo • If the current commit already has a tag, use that instead • git tag --contains HEAD • deploy-2011-08-03_14-37-38 • strftime(%Y-%m-%d_%H-%M-%S)Wednesday, September 7, 2011
  30. 30. Sync • Move the code from the local to the remote box • Uses rsync to put it into the remote service directory • Also places a copy of the synced code on the admin boxWednesday, September 7, 2011
  31. 31. Install • Make the code the active path for code on the machine • This is generally installing code into a virtualenv • Updating the “current” symlink in the service directory • Symlink Django settings file based on environmentWednesday, September 7, 2011
  32. 32. Rollback • When you break things, you need to undo quickly • Reset the repository to the previous deployed tag • git tag | grep deploy| sort -nr |head -2 |tail -1 • Deploy that • Very few moving piecesWednesday, September 7, 2011
  33. 33. Start/Stop/Reload • Allow you to bounce services as part of deployment • Allow reload for services that support itWednesday, September 7, 2011
  34. 34. CLI UI • Have nice wrapper commands that do common tasks • deploy host:web-0 full_deploy:airship ➡ pull, build, tag, sync, install • deploy host:web-1 deploy:airship ➡ tag, sync, install • deploy host:web-2 sync:airship ➡ syncWednesday, September 7, 2011
  35. 35. UI cont. • deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
  36. 36. #!/bin/bash cd ~/airdeploy DATE=$(date +%Y_%-m_%-d-%H-%m-%s) echo "deploy" $@ > logs/$DATE.log fab $@ cd - > /dev/null 2>&1Wednesday, September 7, 2011
  37. 37. Meta-commands • Hard-code the correct deployment behavior • “Make easy things easy, and wrong things hard” • Knows what machine each service is deployed to • deploy airship ➡ deploy pull:airship ➡ deploy type:web deploy:airshipWednesday, September 7, 2011
  38. 38. MagicifyingWednesday, September 7, 2011
  39. 39. Magicifying • Now that we have a solid base, we can automate on top • When you do a meta deploy, it should be a “smart deploy”Wednesday, September 7, 2011
  40. 40. Workflow • Deploy to one web server, preferably with one worker • Restart it • Run it against heuristics to determine if it’s broken • If it’s broken, rollback, otherwise continue onWednesday, September 7, 2011
  41. 41. Heuristics • Any 500s • Number of 200s to non-200s • Number of 500s to 200s • Requests a second • Response time • $$$ (Business metrics)Wednesday, September 7, 2011
  42. 42. How it works • Tell load balancer to take machine out of pool • /take_me_out_of_the_lb -> 200 • Start your code with 1 worker and a different port • supervisorctl start canary • Expose metrics from your services over json • Make sure your load balancer weights it appropriately • Poll your metrics for X time before considering it functionalWednesday, September 7, 2011
  43. 43. Thanks • Alex Kritikos • Erik Onnen • SchmichaelWednesday, September 7, 2011
  44. 44. Questions? • Eric Holscher • Urban Airship (Hiring and whatnot) • eric@ericholscher.comWednesday, September 7, 2011

    Als Erste(r) kommentieren

    Loggen Sie sich ein, um Kommentare anzuzeigen.

  • scorpion032

    Sep. 11, 2011
  • streetcleaner

    Sep. 21, 2011
  • lakshminp

    Oct. 30, 2011
  • robertmchardy

    Nov. 2, 2011
  • markkaplan

    Nov. 13, 2011

Talk on how we do deployment at Urban Airship from Djangocon 2011.

Aufrufe

Aufrufe insgesamt

2.496

Auf Slideshare

0

Aus Einbettungen

0

Anzahl der Einbettungen

195

Befehle

Downloads

47

Geteilt

0

Kommentare

0

Likes

5

×