Retrofitting Continuous Delivery
•Download as PPTX, PDF•
2 likes•642 views
Talk @ NYT TimesOpen 7/20/2014
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Retrofitting Continuous Delivery
Similar to Retrofitting Continuous Delivery (20)
Recently uploaded
Recently uploaded (20)
Retrofitting Continuous Delivery
- 6. betterment : investing :: devops : engineering
- 11. Look at the Mess You’ve Made ● Overburdened, inflexible infrastructure from Rackspace ● Only one pre-prod environment ○ And it runs five other things including Jira/Confluence/etc. ● Building and deploying artifacts manually, ad-hoc, as needed ○ “Why is your trunk .war 23 bytes larger than mine?” ● No Automated Functional Testing ● Deployments require downtime. ● Deployments happen at 3am ET ● Manual Provisioning on persistent servers ○ “This wordpress plugin needs php5.3 but we have 5.1” ● Branching (and merging) with Subversion requires significant effort ● …
- 12. yuck. so what’s the good news?
- 13. we shipped a ton of functionality.
- 14. delivery speed faster than innovation speed
- 15. Phase 2: Addicted to Automation
- 16. stop building features. fix what’s painful.
- 17. DevOp s
- 18. fix onboarding.
- 19. boxen. github // @jbarnette + @wfarr
- 20. mac osx + homebrew + puppet + facter
- 23. fix tooling.
- 24. >
- 25. $ time ant clean test deployment 19m 22s $ time gradle clean test intTest war zipJar 2m 36s (~builds/day · Δt · devs · man_days/mo)/min = (6 · 17 · 19 · 20) / 60 = 646 hrs/mo
- 26. fix notifications & logging.
- 28. Whoa.
- 31. everybody loves jenkins build open pull requests* build develop/stage/master cobertura reports coverage * https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin
- 34. AWS + Rackspace (sitting in a tree, N-E-T-W-O-R-K-I-N-G)
- 36. <10ms !
- 37. “canary” ec2 webapps exposed via route53 weighting
- 39. Ansible (not manual) AMIs ● Playbooks are YML that execute over SSH ● Playbooks define necessary machine config o Results in AMI creation in AWS o Returns AMI_ID ● Commit playbooks and current AMI_ID ● Jenkins reads AMI_ID from build artifact o Spins-up necessary EC2 instances o Runs deploy playbook, unrelated to provisioning playbook ● If you need to modify a playbook o Run playbook to create new AMI_ID o Commit new playbook and AMI_ID to repo.
- 41. everybody loves jenkins again deploys to rackspace deploys to ec2 bash or ansible playbooks
- 44. Flyway Database Migrations ● Looks a lot like rake db:migrate o core/src/main/resources/db/migrations o core/src/main/resources/db/cleanups ● Kept track of in a schema_versions table. ● Cleanups are promoted to migrations when develop moves to stage. ● Run as part of weekly releases. ● http://flywaydb.org/
- 45. Database Subsetting with Jailer ● Walks the constraint graph from the mysql schema. ● Requires some manual intervention to cut off circular or “eager” relations. ● Exports a referentially valid subset of data. o User, Account(s), Trade(s), Etc. ● http://jailer.sourceforge.net/
- 46. Users. Duh.
- 47. + = freshdb subsets live in s3. download and run flyway. instant db on any branch.
- 48. eos:better-core(develop)$ ./fresh -f Forcing download... Downloading fresh data... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.0M 100 21.0M 0 0 1004k 0 0:00:21 0:00:21 --:--:-- 1365k :setupFreshDb :setupFreshDb:complete __ _ / _| |_ ___ ____ _ _ _ | |_| | | | / / / _` | | | | | _| | |_| | V V / (_| | |_| | |_| |_|__, | _/_/ __,_|__, | |___/ by fido |___/ [stderr] INFO AbstractCLI.doMain(287) | cli_started [stderr] INFO FlywayDbMigrationCLI.executeBeforeSpring(164) | jdbc:mysql://127.0.0.1:3306/bettermentdbfresh [stderr] INFO FlywayWrapper.migrateTo(154) | Running ALL pending migrations ... [stderr] INFO ApacheCommonsLog.info(43) | Validated 146 migrations (execution time 00:00.616s) [stderr] INFO ApacheCommonsLog.info(43) | Current version of schema `bettermentdbfresh`: 20140617123653 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132438 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132439 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132440 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132441 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140625091709 ...
- 49. ah. that’s better. now what?
- 50. Phase 3: Innovation That Scales
- 51. innovating is easy. demoing is easy.
- 52. product teams move faster than infrastructure teams.
- 53. jruby on rails factory girl for Java multiple schemas webservices
- 54. delivery speed slower than innovation speed
- 55. DevOps
- 56. DevOps
- 58. every team builds product features with delivery in mind
- 59. Questions? first one: can you start at phase 3?
Editor's Notes
- 3+ years @ Betterment Lead Engineer Built some fun stuff @ Fintech Hackathons Been writing code for a while.
- Investing for everyone, if you want to know (or think you know) what you’re doing.
- multiple goals, multiple allocations, subsection of your account, all managed with the same finesse and care, whether you have $10 or $200,000.
- we show you exactly what’s going on, if you want to see it. automatic rebalancing tax loss harvesting
- very, very confusing at times and have the potential to waste a lot of your time before you figure out the right approach the dream of devops is to create an environment where the path of least resistance also yields the most efficient, sane result. betterment is totally onboard with that mission from the ground up.
- 2010, Java6, Spring MVC + Hibernate, Adobe Flex, Wordpress, MySQL Deployment was VERY manual and very infrequent Build Locally, Unpack War, Like playing Minecraft Properties Management - Secrets OTB. Sticky session management w/ JSESSIONID Necessitates 2am deployments Product features rule EVERYTHING. A quick look @ betterment archetecture. Deployment Process keeps up with Rate of Change Fourth S: Slow Enough
- This is what the site looked like when I signed up for Betterment.
- not a pretty picture, in more ways than one…
- No Persistent Sessions
- Git Flow Predictable Releases Boxen Trello Jenkins + Bash Ansible Brochure AWS AWS RDS RSpec + Capybara WebKit What sort of Innovation(s)? JRuby on Rails + Java Services Jailer + Flyway
- Git Flow Predictable Releases Boxen Trello Jenkins + Bash Ansible Brochure AWS AWS RDS RSpec + Capybara WebKit What sort of Innovation(s)? JRuby on Rails + Java Services Jailer + Flyway
- assemble a strikeforce / build a team
- Build a team to clean up the mess that you made getting the product out there. Make the easy way, the right way.I'd love to have rackspace mount it for us and outline the steps required to do so.
- assemble a strikeforce / build a team
- assemble a strikeforce / build a team
- assemble a strikeforce / build a team
- talk about
- assemble a strikeforce / build a team
- release code reliably, even through trading.
- automatically run tests and resolve dependencies every time code changes
- assemble a strikeforce / build a team
- Development Environments
- Development Environments
- assemble a strikeforce / build a team
- assemble a strikeforce / build a team
- assemble a strikeforce / build a team
- automatically run tests and resolve dependencies every time code changes
- assemble a strikeforce / build a team
- can run app against a remote database, but then the process of switching from local to remote is non-trivial. we haven’t gotten the “development on vm” down yet.
- central authority / decentralized knowledge. push built products out to the team
- lay the railroad track but don’t drive the train.