2. Continuous Delivery : Theory
ed
at
build / deploy / feedback
10
20
30
40
50
60
a
om
ut
have idea
implement idea
measure effectiveness
refine idea
GOTO 10
PROFIT
3. Continuous Delivery : Reality
● Everyone is doing it (apparently)
○ Amazon, Etsy, Facebook, Netflix
■ N deploys per day / min (where N > 0)
○ Great idea (in theory)
● What about me?
○ Existing codebase (smells and all)
○ Existing systems & environments (no access)
○ Existing team and politics (no idea)
e
in
ia
rt
4. Continuous Delivery : Practice
Suck less tomorrow!
● Focus on the journey
○ it will be ongoing anyway
○ simplify / automate
● Crawl / Walk / Run
○ Find a place to start
■ most painful
■ simplest entry point
5. Build : Theory
● Dependencies : what we need
lib of Jars / Maven / Ivy
● Artifacts : what we produce
tgz / jar / war
● Reproducible : scripted
Ant / Maven / Gradle
● Automatable
Jenkins
6. Build : Reality
● GOOD
○ Using ant + ivy + sh + junit
● NOT SO GOOD
○
○
○
○
○
Issues with build dependencies
No output artifact
No easy way to create DB
Only worked on developers machine
Took a day or two to get new team members working
7. Build : Practice
checkout / build / run
● Tidy up existing ant build (based on intent)
○ clean / compile / test / package
○ create / bootstrap DB locally
● Bundle build properties
● Run app locally
● Generate reports
○ Unit tests (JUnit)
○ Static analysis (FindBugs et al)
8. Deploy : Theory
● Push button / Zero downtime
○ Not always easy: Java != rails != php/apache
● Deploy from developers machine is FAIL
○ Shared env or jump box
○ Then automate (via Jenkins)
● What version is running where
○ Build page
○ All servers up to date with latest version
9. Deploy : Reality
● GOOD
○ Scripted (somewhat)
● NOT SO GOOD
○
○
○
○
○
Only from developers machine [FAIL!]
Ignored errors (FULL STEAM AHEAD)
Rsync of build outcome
Manual stop / start
Customers may see errors during rollout
10. Deploy : Practice
● bash script (Capistrano or similar)
○ stop / upload / unpack / restart
○ similar structure to capistrano on filesystem
● Verify
○ Build page with stats (bundled in artifact)
● DB changes
○ Backwards compatible changes (process)
○ Liquibase is great
●
Automated (via Jenkins of course)
11. Run : Theory
● Elastic scaling
○ Dynamic provisioning
○ Automated service management
● Automated monitoring and alerting
○ Sensu / Nagios / ganglia
○ LogStash / GrayLog2
● Dashboards for EVERYTHING!
12. Run : Reality
● ssh to servers
○ to start stop services
○ monitor perf via top and ps
○ grep and tail -f of log files
● Only ‘qualified’ admins allowed
● No business metrics (manual reports)
● No visibility
● RESTART
!
G
N
I
N
R
A
W
Prone to human ERORS
13. Run : Practice
● New Relic
○ servers for free
○ app with paid
○ great insight
● Centralized logging
○ rsyslog:unification/collection
○ papertrail:aggregation & display
14. Run : Practice
● Health Metrics (in app)
○ home rolled exposed via web
○ Metrics by CodaHale / Pingdom
● Biz Metrics
○ DucksBoard
■ simple funnel
■ attrition and activity
● Monit
○ alert and restart (if needed)
15. Environments : Theory
‘Works on my machine’
● Scale – vertically or horizontally
● automatable, reproducible
○ Green / Blue deploys FTW
● Chef / Puppet / Ansible ... just pick one
● If a server fails, can you rebuild it?
● If an environment fails, can you rebuild it?
16. Environments : Reality
You won’t always know what you inherit!
●
●
●
●
●
Private cloud hosting
Limited upgrade ability
Manual updates
Uncertain/misunderstood foundation
SSH and rsync to update and deploy
○ overwrote existing codebase
17. Environments : Practice
● Picked Chef
(Vi vs Emacs …)
○ Roles for baseline / Web / App / DB nodes
○ Environments for DEV / STG / PRD
● Vagrant for testing recipes locally
○ config checked into app project
● Automate (via Jenkins of course)
○ validate cookbooks via FoodCritic
○ upload to hosted chef
○ rollout / reprovision via bash / ssh
18. Environments : Practice
● DB backup / restore
○ do it and test it regularly
○ scripted
○ hire an expert
Delegate to others
● Mandrill for outgoing SMTP
● DNSimple for managed DNS
● Pingdom for simple uptime
○ custom health check endpoint
19. Environment : Current State
Rackspace Cloud
Hosted chef
Managed by Jenkins
○ Re-Create dev / staging / production
○ deploy staging / production
Monitored via
○ Pingdom
○ New Relic
○ PaperTrail app
Rolled out to iOS app dev TOO!
○ Xcode CLI / OCUnit / Testflight via Jenkins
20. OH: I deployed to production,
nobody noticed and nothing
went wrong
FTW!
