Today’s cutting edge companies have software release cycles measured in days instead of months. This agility is enabled by the DevOps practice of continuous delivery, which automates building, testing, and deploying all code changes. This automation helps you catch bugs sooner and accelerates developer productivity. In this session, we’ll share best practices (including ones followed internally at Amazon) and how you can bring them to your company by using open source and AWS services.
Speaker: Raghuraman Balachandran, Solutions Architect, Amazon India
3. Software moves faster today
Software creation and distribution is
easier and faster than ever:
• Startups can now take on giants with little to
no funding ahead of time
• Getting your software into the hands of
millions is a download away
• Your ability to move fast is paramount to your
ability to fight off disruption
4. What tools do you need to move fast?
Releasing software in this new software-driven world
requires a number of tools:
• Tools to manage the flow of your software development
release process
• Tools to properly test and inspect your code for defects
and potential issues
• Tools to deploy your applications
5. First, we need to
understand a little bit
about software release
processes
https://www.flickr.com/photos/jurvetson/5201796697/
6. • Integration
tests with
other systems
• Load testing
• UI tests
• Penetration
testing
Release processes have four major phases
Source Build Test Production
• Check-in
source code
such as .java
files.
• Peer review
new code
• Compile code
• Unit tests
• Style checkers
• Code metrics
• Create
container
images
• Deployment
to production
environments
10. DevOps: Culture + Practices + Tools
Each 2-pizza team “owns” their
product:
• Creates product (software
typically)
• Handles Q/A of that product
• Responds to issues, is on-call
• “you build it, you run it”
• Supports service & tracks/goals
against business and technical
metrics
https://secure.flickr.com/photos/lox/9408028555
11. DevOps: Culture + Practices + Tools
Each 2-pizza team’s practices
largely open so far as standards are
met:
• Agile? Scrum? Daily standups?
Weekly? None? Whatever you
works for your team!
• No centralized change
management
board/team/approval, but tools
that require a degree of sign-
off/process review
https://secure.flickr.com/photos/lox/9408028555
12. DevOps: Culture + Practices + Tools
Each 2-pizza team developers given
a “box of tools”:
• Use these or operationalize your
own:
• Time spent on operations is less
time to spend on development
• Less time spent on development
is increased risk of missing goals
• Tools provide “guard rails” that
enforce best/better practices
• Tools maintained by other 2-pizza
teams
https://secure.flickr.com/photos/lox/9408028555
13. In 2009, we
ran a study to
find out where
inefficiencies
might still exist
14. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to
Prod
15. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to
Prod
Mins Days Mins Days Mins Days Mins
16. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to
Prod
Weeks
Mins Days Mins Days Mins Days Mins
17. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to
Prod
Weeks
Mins Days Mins Days Mins Days Mins
18. We built tools to
automate our software
release process
https://secure.flickr.com/photos/lindseygee/5894617854/
21. We move pretty fast at Amazon:
In 2014:
• Thousands of service teams across Amazon
• + Building microservices
• + Practicing continuous delivery
• + Many environments (staging, beta, production,
multiple regions)
=50 million deploys
22. Every year we survey our
software developers and in 2014
results found only one
development tool/service could
be correlated statistically with
happier developers:
Our pipelines service!
33. AWS Code Services
Source Build Test Production
Third Party
Tooling
Software Release Steps:
AWS CodeCommit AWS CodeBuild AWS CodeDeploy
AWS CodePipeline
34. Secure, scalable, and managed Git source
control
Use standard Git tools
Scalability, availability, and durability of
Amazon S3
Encryption at rest with customer-specific keys
No repo size limit
Post commit hooks to call out to SNS/Lambda
AWS CodeCommit
35. Source control in the cloud
Secure Fully
managed
High
availability
Store
anything
36. AWS CodeCommit
git pull/push CodeCommit
Git objects in
Amazon S3
Git index in
Amazon
DynamoDB
Encryption key
in AWS KMS
SSH or HTTPS
38. Build & test your
application
https://secure.flickr.com/photos/spenceyc/7481166880
39. Fully managed build service that compiles source code,
runs tests, and produces software packages
Scales continuously and processes multiple builds
concurrently
You can provide custom build environments suited to
your needs via Docker images
Only pay by the minute for the compute resources you
use
Launched with CodePipeline and Jenkins integration
AWS CodeBuild
40. How does it work?
1. Downloads source code
2. Executes commands configured in the buildspec in
temporary compute containers (created fresh on every
build)
3. Streams the build output to the service console and
CloudWatch logs
4. Uploads the generated artifact to an S3 bucket
41. How can I automate my release process with CodeBuild?
• Integrated with AWS CodePipeline for CI/CD
• Easily pluggable (API/CLI driven)
• Bring your own build environments
• Create Docker images containing tools you need
• Open source Jenkins plugin
• Use CodeBuild as the workers off of a Jenkins master
42. buildspec.yml Example
version: 0.1
environment_variables:
plaintext:
JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64"
phases:
install:
commands:
- apt-get update -y
- apt-get install -y maven
pre_build:
commands:
- echo Nothing to do in the pre_build phase...
build:
commands:
- echo Build started on `date`
- mvn install
post_build:
commands:
- echo Build completed on `date`
artifacts:
type: zip
files:
- target/messageUtil-1.0.jar
discard-paths: yes
43. buildspec.yml Example
version: 0.1
environment_variables:
plaintext:
JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64"
phases:
install:
commands:
- apt-get update -y
- apt-get install -y maven
pre_build:
commands:
- echo Nothing to do in the pre_build phase...
build:
commands:
- echo Build started on `date`
- mvn install
post_build:
commands:
- echo Build completed on `date`
artifacts:
type: zip
files:
- target/messageUtil-1.0.jar
discard-paths: yes
• Variables to be used by phases of
build
• Examples for what you can do in
the phases of a build:
• You can install packages or run
commands to prepare your
environment in ”install”.
• Run syntax checking,
commands in “pre_build”.
• Execute your build
tool/command in “build”
• Test your app further or ship a
container image to a repository
in post_build
• Create and store an artifact in S3
44. Building Your Code
“Building” code typically refers to languages that
require compiled binaries:
• .NET languages: C#, F#, VB.net, etc.
• Java and JVM languages: Java, Scala,
JRuby
• Go
• iOS languages: Swift, Objective-C
We also refer to the process of creating Docker
container images as “building” the image. EC2
45. No Building Required!
Many languages don’t require building. These
are considered interpreted languages:
• PHP
• Ruby
• Python
• Node.js
You can just deploy your code!
EC2
47. Automates code deployments to any instance
Handles the complexity of updating your
applications
Avoid downtime during application deployment
Rollback automatically if failure detected
Deploy to Amazon EC2 or on-premises
servers, in any language and on any operating
system
Integrates with third-party tools and AWS
AWS CodeDeploy
49. appspec.yml Example
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
permissions:
- object: /var/www/html
pattern: “*.html”
owner: root
group: root
mode: 755
hooks:
ApplicationStop:
- location: scripts/deregister_from_elb.sh
BeforeInstall:
- location: scripts/install_dependencies.sh
ApplicationStart:
- location: scripts/start_httpd.sh
ValidateService:
- location: scripts/test_site.sh
- location: scripts/register_with_elb.sh
• Remove/add instance to ELB
• Install dependency packages
• Start Apache
• Confirm successful deploy
• More!
• Send application files to one
directory and configuration
files to another
• Set specific permissions on
specific directories & files
50. v2 v2 v2 v2 v2 v2
one at a time
half at a time
all at once
v2 v2 v2 v1 v1 v1
v2 v1 v1 v1 v1 v1 Agent Agent
Dev Deployment group
OR
Prod Deployment group
Agent
AgentAgent
Agent Agent
Agent
Choose Deployment Speed and Group
52. Continuous delivery service for fast and
reliable application updates
Model and visualize your software release
process
Builds, tests, and deploys your code every time
there is a code change
Integrates with third-party tools and AWS
AWS CodePipeline
- this talk is broken into 2 sections
- first, I'll start with the backstory about Amazon's own DevOps transformation, and the changes that we made to become more agile with our product delivery
- after covering this history, we're going to switch back to the present
- I'm going to introduce 3 new AWS services that give you the same type of tools that we use internally at Amazon
- You should walk away with a high level understanding of the different parts involved with a DevOps transformation, and an idea of how you could use our AWS Code services in your own DevOps processes
I want to take a moment to talk about different release processes.
Each team’s release process takes a different shape to accommodate the needs of each team.
Nearly all release processes can be simplified down to four stages – source, build, test and production. Each phase of the process provides increase confidence that the code being made available to customers will work in the way that was intended.
During the source phase, developers check changes into a source code repository. Many teams require peer feedback on code changes before shipping code into production. Some teams use code reviews to provide peer feedback on the quality of code change. Others use pair programming as a way to provide real time peer feedback.
During the Build phase an application’s source code is built and the quality of the code is tested on the build machine. The most common type of quality check are automated tests that do not require a server in order to execute and can be initiated from a test harness. Some teams extend their quality tests to include code metrics and style checks. There is an opportunity for automation any time a human is needed to make a decision on the code.
The goal of the test phase is to perform tests that cannot be done on during the build phase and require the software to be deployed to a production like stages. Often these tests include testing integration with other live systems, load testing, UI testing and penetration testing. At Amazon we have many different pre-production stages we deploy to. A common pattern is for engineers to deploy builds to a personal development stage where an engineer can poke and prod their software running in a mini prod like stage to check that their automated tests are working correctly. Teams deploy to pre-production stages where their application interacts with other systems to ensure that the newly changed software work in an integrated environment.
Finally code gets deployed to production. Different teams have different deployment strategies though we all share a goal of reducing risk when deploying new changes and minimizing the impact if a bad change does get out to production.
Each of these steps can be automated without the entire release process being automated. There are several levels of release automation that I’ll step through.
Continuous Integration
Continuous Integration is the practice of checking in your code to the continuously and verifying each change with an automated build and test process. Over the past 10 years Continuous Integration has gained popularity in the software community. In the past developers were working in isolation for an extended period of time and only attempting to merge their changes into the mainline of their code once their feature was completed. Batching up changes to merge back into the mainline made not only merging the business logic hard, but it also made merging the test logic difficult. Continuous Integration practices have made teams more productive and allowed them to develop new features faster. Continuous Integration requires teams to write automated tests which, as we learned, improve the quality of the software being released and reduce the time it takes to validate that the new version of the software is good.
There are different definitions of Continuous Integration, but the one we hear from our customers is that CI stops at the build stage, so I’m going to use that definition.
Continuous Delivery
Continuous Delivery extends Continuous Integration to include testing out to production-like stages and running verification testing against those deployments. Continuous Delivery may extend all the way to a production deployment, but they have some form of manual intervention between a code check-in and when that code is available for customers to use.
Continuous Delivery is a big step forward over Continuous Integration allowing teams to be gain a greater level of certainty that their software will work in production.
Continuous Deployment
Continuous Deployment extends continuous delivery and is the automated release of software to customers from check in through to production without human intervention. Many of the teams at Amazon have reached a state of continuous deployment. Continuous Deployment reduces the time for your customers to get value from the code your team has just written, with the team getting faster feedback on the changes you’ve made. This fast customer feedback loop allow you to iterate quickly, allowing you to deliver more valuable software to your customers, quicker.
So we conducted a study. “Understanding Amazon’s Software Development Process Through Data” in 2009 by John Rauser
We wanted to find out how long it takes to go from code check-in through to code being available in production. This included the time it took to build, test and deploy our software.
We learned that this was taking a long time. In the order of weeks. And we didn’t want it take weeks to get a code change out to production.
What we did discover was our processes had lot of human, manual work in them which were taking most of the time. Developers would use tickets or emails to track their release process. Developers would ticket or email other developers to run a build at which point a bunch of requests would batch up before being run. Once the build was done, new tickets were cut to deploy their software. Those requests may also batch up, increasing the time it took for a change to reach production.
What we did discover was our processes had lot of human, manual work in them which were taking most of the time. Developers would use tickets or emails to track their release process. Developers would ticket or email other developers to run a build at which point a bunch of requests would batch up before being run. Once the build was done, new tickets were cut to deploy their software. Those requests may also batch up, increasing the time it took for a change to reach production.
This was the problem we needed to solve. We needed to automate the production line of developer work so that humans were not longer causing developers to wait, when that work could be automated away.
This was the problem we needed to solve. We needed to automate the production line of developer work so that humans were not longer causing developers to wait, when that work could be automated away.
https://www.flickr.com/photos/stevendepolo/5749192025/
- after we tell customers the story of our DevOps transformation, they typically ask us how they can do the same
- I'm not going to over-simplify this, because it is a very complex answer
- this can involve organizational changes, cultural changes, and process changes
- plus there's no one right answer for these
- every company is going to tweak their approach to optimize for their own environment
- but there is one standard thing that every DevOps transformation needs, and that's an efficient and reliable continuous delivery pipeline
- that's the focus for the rest of this talk
Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example.
Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket.
Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage.
In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction
Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage
The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example.
Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket.
Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage.
In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction
Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage
The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example.
Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket.
Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage.
In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction
Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage
The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example.
Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket.
Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage.
In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction
Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage
The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- I demonstrated using a few partner solutions with our AWS Code services
- Here's the full list of partners who have integrated their tools with CodePipeline and CodeDeploy
- And this list is growing as we welcome more integrations into our tools suite
- Many of these partners have booths in the expo hall
- I encourage everyone to explore their solutions to see how they might benefit your cloud development projects
- We even have some partners here at this talk, so when we take Q&A (up front|| out in the hallway) after the talk, you'll be able to ask questions of them as well