DevOps On AWS - Deep Dive on Continuous Delivery
•Als PPTX, PDF herunterladen•
1 gefällt mir•254 views
Deep dive on DevOps, continuous delivery pipelines, and AWS tools that help to accelerate software delivery process
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie DevOps On AWS - Deep Dive on Continuous Delivery
Ähnlich wie DevOps On AWS - Deep Dive on Continuous Delivery (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
DevOps On AWS - Deep Dive on Continuous Delivery
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mikhail Prudnikov, Solutions Architect September 2016 DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
- 3. Software moves faster today Software creation and distribution is easier and faster than ever: • Startups can now take on giants with little to no funding ahead of time • Getting your software into the hands of millions is a download away • Your ability to move fast is paramount to your ability to fight off disruption
- 4. Old software delivery model The software delivery model has drastically changed New software delivery model
- 5. What tools do you need to move fast? Releasing software in this new software driven world requires a number of things: • Tools to manage the flow of your software development release process • Tools to properly test and inspect your code for defects and potential issues • Tools to deploy your applications
- 6. First, we need to understand a little bit about software release processes https://www.flickr.com/photos/jurvetson/5201796697/
- 7. • Integration tests with other systems • Load testing • UI tests • Penetration testing Release processes have four major phases Source Build Test Production • Check-in source code such as .java files. • Peer review new code • Compile code • Unit tests • Style checkers • Code metrics • Create container images • Deployment to production environments
- 8. Release processes levels Source Build Test Production Continuous integration Continuous delivery Continuous deployment
- 9. Release Processes levels Source Build Test Production Continuous integration Continuous deployment Our focus today Continuous delivery
- 10. Continuous Delivery Benefits Improve developer productivity Find and address bugs quickly Deliver updates fasterAutomate the software release process
- 11. A look back at development at Amazon.. https://secure.flickr.com/photos/pixelthing/15806918992/
- 12. 2001 Development transformation at Amazon: 2001-2009 2009 monolithic application + teams microservices + 2 pizza teams
- 13. Things went much better under this model and teams were releasing faster than ever, but we felt that we could still improve.
- 14. In 2009, we ran a study to find out where inefficiencies might still exist
- 15. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod
- 16. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod Mins Days Mins Days Mins Days Mins
- 17. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod Weeks Mins Days Mins Days Mins Days Mins
- 18. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod Weeks Mins Days Mins Days Mins Days Mins
- 19. We built tools to automate our software release process https://secure.flickr.com/photos/lindseygee/5894617854/
- 20. Automated actions and transitions; from check- in to production Development benefits: • Faster • Safer • Simplification & standardization • Visualization of the process Pipelines
- 21. This has continued to work out really well: In 2014: • Thousands of service teams across Amazon • Building microservices • Practicing continuous delivery • Many environments (staging, beta, production) 50 million deploys
- 22. We continue to survey our software developers every year and in 2014 results found only one development tool/service could be correlated statistically with happier developers: Our pipelines service!
- 24. Continuous delivery service for fast and reliable application updates Model and visualize your software release process Builds, tests, and deploys your code every time there is a code change Integrates with 3rd party tools and AWS AWS CodePipeline
- 25. AWS CodePipeline Benefits Configurable workflow Easy to integrate Improved quality Rapid delivery Get started fast
- 29. Build JenkinsOnEC2 Jenkins Staging-Deploy JavaApp Elastic Beanstalk Prod-Deploy JavaApp Elastic Beanstalk QATeamReview Manual Approval CodePipeline MyApplication Manual Approvals Review
- 30. You can add a manual approval at the point where you want the pipeline to stop running until someone approves or rejects the revision in progress. Manual Approvals – New! • Pipeline will stop executing when it has reached the point at which you set the approval action. • Pipeline execution resumes only when the action has been approved • Approval action managed with AWS Identity and Access Management (IAM) permissions • Notify approvers in several ways including email, SMS, webhooks, and more • Useful for manual QA actions or as party of “Canary” deploy models
- 31. 8. Retrieve build artifact EC2 instance CodePipeline Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk Source Artifact S3 Build Artifact S3 5. Get source artifact 1. Get Changes 6. Store build artifact 3. Poll for Job 4. Acknowledge Job 7. Put Success 9. Deploy build artifact Elastic Beanstalk Web container Java App MyApplication
- 32. We have a strong partner list, and it’s growing Source Build Test Deploy *beta
- 33. AWS service integrations Source Invoke Logic Deploy AWS Elastic Beanstalk Amazon S3 AWS CodeDeployAWS Lambda AWS CodeCommit AWS OpsWorks
- 34. Building your application development release pipeline https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
- 35. DEMO!
- 38. Build & test your application https://secure.flickr.com/photos/spenceyc/7481166880
- 39. Building Your Code “Building” code typically refers to languages that require compiled binaries: • .NET languages: C#, F#, VB.net, etc. • Java and related languages: Java, Scala, JRuby • Go • iOS languages: Swift, Objective-C We also refer to the process of creating Docker container images as “building” the image. EC2
- 40. No Building Required! Many languages don’t require building. These are considered interpreted languages: • PHP • Ruby • Python • Node.js You can just deploy your code! EC2
- 41. Testing Your Code Testing is both a science and an art form! Goals for testing your code: • Want to confirm desired functionality • Catch programming syntax errors • Standardize code patterns and format • Reduce bugs due to non-desired application usage and logic failures • Make applications more secure
- 42. Where to Start with Testing 1. Start with a generic syntax linter • Are you missing brackets, commas, etc.? 2. Work on the top of the pyramid initially to test user facing experience/functionality • Are your pages rendering properly? 3. Move on to deeper UI/service checks • Are these two components in sync? 4. Start unit tests against basic user functions and move deeper into application logic • Invest more time and effort here as you find bugs or failures in production
- 44. Automates code deployments to any instance Handles the complexity of updating your applications Avoid downtime during application deployment Deploy to Amazon EC2 or on-premises servers, in any language and on any operating system Integrates with 3rd party tools and AWS AWS CodeDeploy
- 45. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh
- 46. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh • Remove/add instance to ELB • Install dependency packages • Start Apache • Confirm successful deploy • More! • Send application files to one directory and configuration files to another • Set specific permissions on specific directories & files
- 47. v2 v2 v2 v2 v2 v2 one at a time half at a time all at once v2 v2 v2 v1 v1 v1 v2 v1 v1 v1 v1 v1 Agent Agent Dev Deployment group OR Prod Deployment group Agent AgentAgent Agent Agent Agent Choose Deployment Speed and Group
- 49. Launching to Production After you’ve built and tested your code and hopefully gone through a few pre-production deploys, its time for the real thing! You’ll want think about: • Impact to customers • Impact to infrastructure • Impact to business How can we track these and communicate deploys?
- 50. Extend AWS CodePipeline Using Custom Actions Update tickets Provision resources Update dashboards Mobile testing Send notifications Security scan
- 51. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy Action Custom Action JavaApp Elastic Beanstalk MyApplication CodePipeline Deploy JavaApp Elastic Beanstalk Job Worker 3. Perform Job 1. Poll for Job 2. Acknowledge Job 4. Put Success Amazon EC2 With custom actions, the job worker drives the interaction between AWS CodePipeline and other applications or services
- 52. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy MyAction AWS Lambda JavaApp Elastic Beanstalk 2. Perform Job 1. Invoke Lambda function 3. PutJobSuccessResult MyApplication CodePipeline With AWS Lambda-based actions, AWS CodePipeline drives the integration with Lambda, which then connects with other applications or services AWS Lambda 3. PutJobSuccessResult w/ Continuation Token 4. Invoke Lambda function w/ Continuation Token 5. PutJobSuccessResult #3 & #4 repeat until no continuation token is sent, signaling the action has been completed (#5).
- 53. What Extension Method Should I Use? Lambda Custom Action Short-running tasks are easy to build Can perform any type of workload Long-running tasks need more work Control over links displayed in console Node.js, Python, and Java support Any language support Runs on AWS Can run on-premises No servers to provision or manage Requires compute resources
- 54. FIN, ACK We’ve seen a quick run through today of the benefits of continuous delivery on our software release process: • Continuous integration (build/test) helps shrink our feedback loop greatly • We can get our software out in front of our users much more rapidly • By moving faster we can actually ensure better quality • CodePipeline allows for integration with almost any service or tool you can think of! • Plus visualization of what’s going on!
- 55. Try it out today Test out CodePipeline and spin up a full continuous delivery pipeline using the Starter Kit bit.ly/AWSCodeStarterKit
- 56. Resources to learn more: • Continuous integration: https://aws.amazon.com/devops/continuous-integration/ • Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/ • CodePipeline • https://aws.amazon.com/codepipeline/ • https://aws.amazon.com/documentation/codepipeline/ • CodeDeploy • https://aws.amazon.com/codedeploy/ • https://aws.amazon.com/documentation/codedeploy/ • https://github.com/awslabs/aws-codedeploy-samples • Code Services Starter Kit: http://bit.ly/AWSCodeStarterKit But wait, there’s more!
- 57. Thank you!
Hinweis der Redaktion
- https://secure.flickr.com/photos/mgifford/4525333972
- https://routingbyrumor.files.wordpress.com/2007/12/compusa.jpg, http://static.giantbomb.com/uploads/original/1/15187/2505806-0028159468-compa.jpg, http://www.blogcdn.com//media/2011/07/applestoresoftware.jpg
- https://www.flickr.com/photos/jurvetson/5201796697/ Need to be damn good at CI/CD aka making the donuts. If you struggle to do CI/CD for a monolith, you will struggle to do it in a complicated microservices environment. The process is a lot more like making donuts, than it is say cars.
- I want to take a moment to talk about different release processes. Each team’s release process takes a different shape to accommodate the needs of each team. Nearly all release processes can be simplified down to four stages – source, build, test and production. Each phase of the process provides increase confidence that the code being made available to customers will work in the way that was intended. During the source phase, developers check changes into a source code repository. Many teams require peer feedback on code changes before shipping code into production. Some teams use code reviews to provide peer feedback on the quality of code change. Others use pair programming as a way to provide real time peer feedback. During the Build phase an application’s source code is built and the quality of the code is tested on the build machine. The most common type of quality check are automated tests that do not require a server in order to execute and can be initiated from a test harness. Some teams extend their quality tests to include code metrics and style checks. There is an opportunity for automation any time a human is needed to make a decision on the code. The goal of the test phase is to perform tests that cannot be done on during the build phase and require the software to be deployed to a production like stages. Often these tests include testing integration with other live systems, load testing, UI testing and penetration testing. At Amazon we have many different pre-production stages we deploy to. A common pattern is for engineers to deploy builds to a personal development stage where an engineer can poke and prod their software running in a mini prod like stage to check that their automated tests are working correctly. Teams deploy to pre-production stages where their application interacts with other systems to ensure that the newly changed software work in an integrated environment. Finally code gets deployed to production. Different teams have different deployment strategies though we all share a goal of reducing risk when deploying new changes and minimizing the impact if a bad change does get out to production. Each of these steps can be automated without the entire release process being automated. There are several levels of release automation that I’ll step through.
- Continuous Integration Continuous Integration is the practice of checking in your code to the continuously and verifying each change with an automated build and test process. Over the past 10 years Continuous Integration has gained popularity in the software community. In the past developers were working in isolation for an extended period of time and only attempting to merge their changes into the mainline of their code once their feature was completed. Batching up changes to merge back into the mainline made not only merging the business logic hard, but it also made merging the test logic difficult. Continuous Integration practices have made teams more productive and allowed them to develop new features faster. Continuous Integration requires teams to write automated tests which, as we learned, improve the quality of the software being released and reduce the time it takes to validate that the new version of the software is good. There are different definitions of Continuous Integration, but the one we hear from our customers is that CI stops at the build stage, so I’m going to use that definition. Continuous Delivery Continuous Delivery extends Continuous Integration to include testing out to production-like stages and running verification testing against those deployments. Continuous Delivery may extend all the way to a production deployment, but they have some form of manual intervention between a code check-in and when that code is available for customers to use. Continuous Delivery is a big step forward over Continuous Integration allowing teams to be gain a greater level of certainty that their software will work in production. Continuous Deployment Continuous Deployment extends continuous delivery and is the automated release of software to customers from check in through to production without human intervention. Many of the teams at Amazon have reached a state of continuous deployment. Continuous Deployment reduces the time for your customers to get value from the code your team has just written, with the team getting faster feedback on the changes you’ve made. This fast customer feedback loop allow you to iterate quickly, allowing you to deliver more valuable software to your customers, quicker.
- https://secure.flickr.com/photos/pixelthing/15806918992/
- So we conducted a study. “Understanding Amazon’s Software Development Process Through Data” in 2009 by John Rauser We wanted to find out how long it takes to go from code check-in through to code being available in production. This included the time it took to build, test and deploy our software. We learned that this was taking a long time. In the order of weeks. And we didn’t want it take weeks to get a code change out to production.
- What we did discover was our processes had lot of human, manual work in them which were taking most of the time. Developers would use tickets or emails to track their release process. Developers would ticket or email other developers to run a build at which point a bunch of requests would batch up before being run. Once the build was done, new tickets were cut to deploy their software. Those requests may also batch up, increasing the time it took for a change to reach production.
- What we did discover was our processes had lot of human, manual work in them which were taking most of the time. Developers would use tickets or emails to track their release process. Developers would ticket or email other developers to run a build at which point a bunch of requests would batch up before being run. Once the build was done, new tickets were cut to deploy their software. Those requests may also batch up, increasing the time it took for a change to reach production.
- This was the problem we needed to solve. We needed to automate the production line of developer work so that humans were not longer causing developers to wait, when that work could be automated away.
- This was the problem we needed to solve. We needed to automate the production line of developer work so that humans were not longer causing developers to wait, when that work could be automated away.
- https://secure.flickr.com/photos/lindseygee/5894617854/
- CodePipeline is also Easy to Integrate with I’ll be showing your how to extend CodePipeline in a few minutes. CodePipeline can help you Improve Quality A standardized release process reduces the chance of customer errors being reaching your customers. Rapid Delivery Automating your release workflow with CodePipeline means that your software is getting into your customers hands faster. You can quickly iterate on customer feedback and get new features to customers, faster. Get Started with CodePipeline is Fast With CodePipeline you can started immediately modelling your release processes. We’re currently available in us-east-1 and us-west-2. There are no servers for you to provision or set up. CodePipeline is a fully managed, continuous delivery service that connects to your existing tools and systems. CodePipeline is available now. You can get started with your first pipeline free and then $1 per month for each additional pipeline.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- We have partnered with popular source, build, test and deployment systems to provide out-of-the-box integrations. Jenkins, CloudBees and Solano offer CI services for build stages BlazeMeter, Apica, HP StormRunner and Runscope are load testing partners. GhostInspector is a User Interface Testing partner GitHub is a source code partner Xebia Labs is a deployment partner.
- https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
- https://secure.flickr.com/photos/spenceyc/7481166880
- We will want to confirm that this is actually our best practices/guidance
- The effort you put into the testing triangle should not be evenly distributed! Many experts in the industry recommend a 70,20,10 mix. (will need sources)
- https://secure.flickr.com/photos/simononly/15386966677
- https://www.flickr.com/photos/spacex/16510243060/
- We’re going to add a new custom action in the deploy stage that will send a message to our messaging App, Slack. The Custom Action will poll for jobs acknowledge the job send a motivational message to our messaging app and then return successfully. This is the high level architecture of what we’re going to build.
- NOTE THIS NEEDS FIXING.
- We’ve had one customer write their custom action in cron and bash.