Continuous Delivery Agiles 2014 Medellin
•Als PPT, PDF herunterladen•
3 gefällt mir•1,086 views
Conference about the Continuous Delivery Project on OLX. Tools we used, tools we created and opensourced like Octopush and lessons learned.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Ähnlich wie Continuous Delivery Agiles 2014 Medellin
Ähnlich wie Continuous Delivery Agiles 2014 Medellin (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Continuous Delivery Agiles 2014 Medellin
- 1. Free Classifieds www.olx.com Continuous Delivery Release Management Buenos Aires, April 2014
- 2. 2 Agenda 1. Initial Situation 2. Initial Setup 3. Continuous Delivery Adoption - How we started 4. Deployment stages 5. Application configuration deployment 6. Introducing Octopush 7. Architecture evolution 8. Tooling 9. Culture changing 10. Lessons learned 11. What we have achieved 12. Next steps 13. Q&A
- 3. 3
- 4. 4 Initial Situation Situation Complication BigBang/Stressful releases, once every two weeks Multiple Development Teams -> # deployments increasing Few automation - Most of the tasks were manual Monolithic platform / Dependency hell Manual Configuration SVN Some teams doing CI Distrust & escepticism • Business Risk Product Delivery Product Quality Need to shorten the time to market when adding new features Competitive advantage: Need more efficient processes and reduce operational costs. • Growing teams • Going Global / distributed team / different time zones
- 5. SVN 5 CI Server RM Server ssh/console Trigger job Dev Team I Release Management Initial CI and RM Setup Dev Team II
- 7. 7 Decision
- 8. 8 Initial Assessment Source: http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model
- 9. Continuous Delivery Adoption • Sharing the situation and necessity • Management commitment • Resources allocated • Continuous Integration • Continuous Delivery • Continuous deployment • Reduce Errors / Lower Stress / Lower Delta Changes Release • On Demand Deployment/ Built Quality In • Everybody Is Responsible for the Delivery Process/ Continuous 9 Improvement • Establish Phases and goals • Scope by component • Assumptions / expectations • Project Risk Identification Commitment Concepts alignment Setting Goals Project Management
- 10. Dev Env Integration/Acceptance Env 10 High Level View Commit Stage Deployment Stage Acceptance Stage Live Production Env Manual gate :: RM Button Automatic :: Web-hook
- 11. 11 Integration Jenkins Commit Stage from Git master branch to Artifactory Git Master/Branch Artifactory Release Jenkins Checkout Publish binaries Trigger Release Checkpoint: - build or pre-compile - run unit test & coverage Deployment Stage [ENV available?]::Go::Wait Commit hook Commit Build
- 12. 12 Deployment Stage Release Jenkins from Integration Jenkins Acceptance Artifactory Git repo Env. [If code] Get Binaries [If config] Get Config from Artifactory to Acceptance Env To Integration Jenkins Run Deploy Script Run smoke test Automatic Rollback upon failure Code OR Config, NEVER both
- 13. 13 Acceptance Stage Integration/Dev Jenkins from Release Jenkins Acceptance Env. from Acceptance Env to Production Gate To Release Jenkins Run Acceptance Tests Run Performance Tests* * Using YSlow for frontend testing Automatic Rollback upon failure
- 14. Application configuration deployment 14 Config files, from the repo to the environment ggiti-tc-cryryppt t
- 15. Architecture evolution Frequency/Complexity/Impact/Risk/Speed deployment 15
- 16. 16 Introducing Octopush Jenkins Mobile Jenkins Frontend Jenkins Backend Jenkins blah Make deployment request Enqueues job Pick next available job Call Jenkins to deploy Polls/notify on completion Deploy Orchestrate Jenkins Hell! OOppeenn S Soouurrccee Call deployment job
- 18. The Revolution will not be centralized 18
- 19. 19 Remaining servers Alert ROLLBACK Wait Monitor Check Canary Release Release pool
- 20. 20 Tooling RREESSTT A APPII
- 22. 23 Lessons learned 1/2 Architecture Smoke Test Infrastructure-as-code Conventions • Application architecture. • Tooling architecture should also be layered and Service oriented. • Integrate your own tools with Jenkins JIRA, IRC, Graphite, most of them have a REST API service. No silver bullet • Divide configuration • Health checks! • Track & version your scripts, follow Software design principles • Same scripts/infra everywhere!, use config files to resolve hostname issues • Scripts should be idempotent • Apps/repos/users/package naming. Canary Releases • Use a Testing pool, automate!.
- 23. 24 Lessons learned 2/2 Communication • Clear channel • IRC • Automate. Strategy • MVP • Advertise progress/delays, achievements, use charts/hard data. • Early adopters.
- 24. Releases Conclusions 25 • Tripled our Production deployment throughput . • More confident and error-free releases. • Less stress, near-zero manual work, easier, faster deploys Achievements Octopush decentraliza tion
- 25. 26 Next steps 1.Native packages 1.Deploy integration with Puppet 1.Virtualization / Docker .
- 26. 27 Q&A .
- 27. Thank You (We are hiring DEVOPS) Continuous Delivery Team Diego Garber @diegarber Google+ Communities: OLX Engineering 28
Hinweis der Redaktion
- Introduce me & OLX, some metrics: 200M users monthly. 40 paises lider.
- The story about CD project, when it started
- Lets just say that developers weren't the most popular guys inside the deparment bubble.
- Multiple Development Team : # deployments increasing : more complex releases. Cultural: Desconfianza, cinismo Product Quality : Not safely deploys Inefficient Processes : not reliable mechanism
- We had many different dev teams working at the same time, some of them had already been working on CI with their own Jenkins Server while some of them weren't yet. The most problematic issue was that inspite of being different teams working on different projects and features, they were sharing the same SVN repo, and that require, as you may imagine, a hell lot of coordination for development and deployment purposes. As you can see in the purple circle, RM had a Jenkins Server of it's own to handle some release tasks, but there were still some scripts that we had to run manually.
- Big queues of DEVs waiting for their deployment, huge amount of workload on Release Management, situation
- Until we found this book which totally changed the way we though and handled releases. So we decided to adopt Continuous Delivery.
- Researching about Continuous Delivery solutions and articles we found this on InfoQ, and it struck us as a very helpful guide to organize and plan our project. This chart basically divides the problem in different tracks (name them) and defines 5 maturity levels for each of them. So the first thing we did is identify where were we standing, and we found that according to this, we were somewhere between base and begginners. And also decided we wanted to get somewhere between Intemediate and advanced by the end of year. It's also important to notice how a level of maturity involves practices improvement in different areas simultaneously, meaning you can just automate your deployments on a monolithic architecture or with no test automation whatsoever.
- Ayuda… consultoria externa, mas gente… Evangelizacion Git => Global Etapas: POC (Prueba de Cpto) Mobile, alineamiento con SOA
- Bosquejo etapas
- Mencionar tema multiples jenkins
- One of the problems we had, as I mentioned before was the app configuration deployment, things like: external services hostnames, feature flags, credentials, etc. We needed to automate without loosing control of the enviromental settings. So we created an extra config repo for every code repo. Each config repo has 3 branches, one for each environment, so the settings values on each of them are slightly different. Developer have read only access over these repos, so when they need to deploy a change, they make a PR, we check the change and if approved, Jenkins takes care of the deployment, thanks to web-hooks. We are using git-crypt to encrypt credential files. After a couple of months we realized we could do even better, by dividing the config between purely environmental settings and the purely application ones. This freed us from having to approve each change when 99% of them were only application-related, this change speeded the process significantly.
- This picture kind of shows what I mentioned before about app architecture moving along with CD. We started with a monolithic code which presented a lot of complexity, impact and Risk on each deployment, this generates fear and distrust which in turn reduces the speed and Frequency of deployments. Where we are now (the low part) we have basically a 3 layered architecture where on the top (presentation) we have the least complex, impactful and risky deployment and at the bottom the exact opposite. So, based on this we're currently deploying much more frequently the presentation layer, approaching the Continuous deployment concept, than the backend: CD.
- Until now what you have seen, solved a lot of our problems, but it created some others. Like for example, having multiple jenkins interacting amongst them didn't give everyone the proper visibility of what deployments were running at any given time, was difficult to follow a pipeline and troubleshoot errors or bottlenecks. But we also wanted to deal with the traffic control issue, we needed a way to stop all deployments at once in a centralized way, or to be able to seriallized or parallelized component deployments according to their dependencies. This is how Octopush was born, it receives deployment request from CI Servers, enqueues and then dispatches them to Jenkins RM in an ordered/intelligent manner.
- Decidimos dar empowerment a la gente y les cedimos el boton de Release a los desarrolladores, esto fue un cambio de paradigma total en nuestro trabajo diario y en la cultura de la empresa. Ahora el dilema era como ceder el control pero al mismo tiempo evitar…
- On the tooling part, we have developed, besides Octopush, a very simple web app, called TagReporter, where we track & show everyone what version of components are deployed on each environment (comp on the rows, env on columns, links takes the user to jira, gihub or jfrog). Additionally we provide information on the Release JIRA ticket that was or is about to be deployed LIVE and its current status. The last column provides different action according to the logged user, devs can create tickets automatically and RE can trigger deployments/rollbacks. The app integrates all this tools via REST API, providing a Transactional deployment service: it calls jenkins, interacts with JIRA and reports to IRC.
- Focused on getting each change right the first time
- Starting with Resistance… Anecdota, quedarse sin trabajo. Beer factor!
- Continuous integration as an initial initiative