Continuous Deployment Practices, with Production, Test and Development Environments Running on AWS

Continuous Deployment Practices, with Production, Test
and Development Environments Running on AWS

Chris Munns, Solutions Architect, Chris Barclay, Senior
Product Manager, and Mike Limcaco, Solutions Architect

This talk:

• Not going to spend a lot of time talking about Continuous
Integration(CI) and Continuous Deployment(CD) philosophy
• Will spend more time talking about how AWS can help you if
practicing CI and CD are your goals
• Examples to get you thinking, but not the only way
• AWS + Open Source solutions

• Continuous Integration
• Continuous Deployment

• Continuous Integration Techniques and tools to
• Continuous Deployment implement continuous
processes of applying quality
control in general small
pieces of effort, applied
frequently, to improve the
quality of software, and to
reduce the time taken to
deliver it.

• Continuous Integration Techniques and tools to
• Continuous Deployment improve the process of
software delivery, resulting in
the ability to rapidly, reliably,
and repeatedly push out
enhancements and bug fixes
to customers at low risk and
with minimal manual
overhead.

• Continuous Integration Getting code from
• Continuous Deployment developers’ brains,
through their fingers,
to production quickly
and efficiently, with
positive results.

Continuous Integration & Deployment on AWS

• Treat infrastructure as code
• Automate the testing/deploy process end to end
• Make sure environments mimic each other as closely as possible
• Use repeatable patterns between environments at a different scale
• Use different cost models where it makes sense
• Simplify and streamline the deploy process
• Let AWS services handle control flows
• Track everything (instance metrics, application metrics, logs)

In today’s infrastructure, everything is code.
From the applications developers are writing, to
your configuration management tools, to things like
CloudFormation templates or scripts that call AWS APIs.

Since Infrastructure is code, let’s treat it like code!
– Not JUST Revision control!
– Make use of bug tracking/ticketing systems
– Peer reviews of changes before they happen
– Establish infrastructure code patterns/designs
– Test infrastructure changes like code changes

Let’s talk about
the journey our
code is going to
take to production

1.Code gets written
2.Code gets tested
3.Code gets deployed
4.Code gets consumed

1.Code gets written
– Someone writes code and commits to revision
control system
– Hooks in revision control, system kicks off CI work

2.Code gets tested
– Unit tests, integration tests, db tests, smoke tests,
UI tests
– “Light green, trap clean” OR GOTO STEP 1

3.Code gets deployed
– Ship out that code

4.Code gets consumed
– Customers use it, love it, victory, profit, vacation in
Bora Bora

1.Software ( tools, services, scripts )
2.Infrastructure Environments ( dev, test, prod )
3.Process ( deploy, monitor, alert, track )

1.Software ( tools, services, scripts )
2.Infrastructure Environments ( dev, test, prod )
3.Process ( deploy, monitor, alert, track )

We need tools to help work with all of the
above quickly and more efficiently

First stop on our journey: Continuous Integration-ville
• Help prove code quality and function repeatedly with predefined results
• Lots of options; self hosted, open source, closed source, and SaaS

Continuous Integration - Jenkins

• Open Source
• Well established and used by many
• Has plugins for EC2/SQS/SNS/CloudFormation!
• Supports spot pricing!
“An extendable open source continuous • Supports the ability to put workers into a
integration server” “standby” mode by stopping instead of
terminating
• Scales well
• Easily add more EC2 instances as workers
• Flexible
Pre-commit • Easy to get started
Hook Internal
CI Workers CI Server Git

Testing Environment Subnet

Test Chef Cookbooks w/ FoodCritic after each Git Commit.

Test Chef Cookbooks w/ FoodCritic after each Git Commit.

Blue dot is good!

Infrastructure Environments
A bad thing people do:

“Developers develop locally on their laptops, mostly OS X based, then deploy to
production, which is Ubuntu. Each laptop has a slightly different setup, and we
don’t maintain software versions across the whole team.”

– Dev and prod not in sync
– Dev not in sync with all of dev
– No testing tier between dev and prod

A bad thing people do:

“Developers develop locally on their laptops, mostly OS X based, then deploy to
production, which is Ubuntu. Each laptop has a slightly different setup, and we
don’t maintain software versions across the whole team.”

– Dev and prod not in sync
– Dev not in sync with all of dev “it worked fine on my laptop”
– No testing tier between dev and prod

NAT Customer
Traffic
Bastion/ Amazon CloudWatch
Chef
RDS DB
Instance Instance
Instance

VPC Subnet VPC Subnet Availability Zone Subnet
VPC VPC Subnet Amazon
SNS

Internet
Amazon
Gateway
Route 53
RDS DB Instance
Standby (Multi-AZ) Instance Instance
ELB ELB

VPC Subnet VPC Subnet VPC Subnet VPC Subnet
Availability Zone

Amazon S3
Amazon
CloudFront

Potential RDS DB Instance Instance
Instance Read
Replica AWS
CloudFormation
Availability Zone
Region

Our Development Infrastructure

Dev MySQL DB Instance DEV APP ELB DEV WEB ELB

Dev Stack Dev Stack
Tier 2 Tier 1

Dev Environment VPC Subnet

NAT Customer
Bastio
Traffic
Amazon CloudWatch
n/Chef
RDS DB
Instance Instance Instance

SNS

Internet
Amazon
Gateway
Route 53
RDS DB Instance
ELB ELB

Availability Zone

Amazon S3
Amazon
CloudFront

Potential Instance Instance
RDS DB
Instance AWS
Read Replica CloudFormation
Availability Zone
Region

Our Development Infrastructure

Amazon S3 Amazon
Amazon
Route 53
CloudFront
Dev DEV APP DEV
Dev Admin
MySQL DB Dev Stack ELB Dev Stack WEB ELB
Instance
Instance Tier 2 Tier 1 Amazon
DynamoDB

VPN
Internet TUNNEL
VPN
Gateway
Developers
Endpoint
&
Operations

NAT Amazon SQS
Instance
VPN facing VPC Subnet

Our Development &Test Infrastructure

Amazon S3 Amazon
Amazon
Route 53
CloudFront
Dev DEV APP DEV
Dev Admin
Instance
DynamoDB

VPN
Internet TUNNEL
VPN
Gateway
Developers
Endpoint
&
Pre-commit Operations
Hook Internal
CI Workers CI Server Git NAT Amazon SQS
Instance
Testing Environment Subnet VPN facing VPC Subnet

• Be prepared to be running multiple environments
– Development
– Testing/QA
– Staging/Pre-prod
– Production
• They should be running as close to the same stack as possible
• Use configuration management and infrastructure orchestration tools
• No one off hosts
• A goal: Go from nothing to fully running instances without human intervention

This all seems like a lot of work,
and potentially costly.

Infrastructure Automation
We want to be able to rapidly stand up environments as we need to.
Sounds like we need some automation tools?

– CloudFormation
– Elastic Beanstalk
– OpsWorks
– Chef
– Puppet

AWS CloudFormation
"WebServer" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, "64" ]},
"SpotPrice" : { "Ref" : "SpotPrice" },
"InstanceType" : m1.large",
"SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }],
"KeyName" : { "Ref" : "KeyName" },
"UserData": {
"Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -vn",
"yum update -y aws-cfn-bootstrapn”,
"curl -L http://www.opscode.com/chef/install.sh | bashn",
"cd /etc/chefn",
"/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/validation.pemn",
"/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/client.rbn",
"/bin/chown -R chef:chef /etc/chefn",

Our Development/Test Infrastructure

Amazon S3 Amazon
Amazon
Route 53
CloudFront
Dev DEV APP DEV
Dev Admin
Instance
DynamoDB

VPN
Internet TUNNEL
VPN
Gateway
Developers
Endpoint
&
Hook Internal
Instance

AWS Elastic Beanstalk & OpsWorks
Elastic Beanstalk:
• Application container framework similar to a PaaS
• Deploy your application into Elastic Beanstalk and it takes care of building a self
healing, auto-scaling, multi-AZ infrastructure
• Allows you to turn some of the knobs under the hood to tweak
• Considered one of the easiest places to start with hosting an application on AWS

OpsWorks:
• Build multi-layer application stacks
• Ties in with Chef for a large degree of flexibility and customization
• Makes deploying applications easier
• More flexible than Elastic Beanstalk, but requires a bit more knowledge

Our Development/Test Infrastructure
Elastic Beanstalk or OpsWorks
Amazon S3 Amazon
Amazon
Route 53
CloudFront
Dev DEV APP DEV
Dev Admin
Instance
DynamoDB

VPN
Internet TUNNEL
VPN
Gateway
Developers
Endpoint
&
Hook Internal
Instance

NAT Customer
Traffic
Elastic Beanstalk or OpsWorks
RDS DB
Instance Instance Instance
Bastion/
Chef
Amazon CloudWatch

SNS

Internet
Amazon
Gateway
Route 53
RDS DB Instance
ELB ELB

Availability Zone

Amazon S3
Amazon
CloudFront

Potential RDS DB Instance Instance
Instance Read
Replica AWS
CloudFormation
Availability Zone
Region

Imagine you had an infrastructure you could turn on/off on
demand, make use of spare capacity at a lower cost,
and/or make a reservation for capacity based on your
usage needs and save money doing so.

Using the Right Cost Model – EC2
• On Demand
• Reserved Instance ( RI ) – 40%+ savings
• Spot – 80%+ savings

Each has its place. For development infrastructure, there
are often places for each:
• On Demand – Developer instances started/stopped daily
• Reserved Instances – Code repository, CI master, DBs
• Spot – CI workers, tiers of dev infrastructure that can tolerate going
away for a bit

RRS S3, CloudFront
Our Development &Test Infrastructure Price Classes,
SPOT/ON-DEMAND DynamoDB RC

Amazon S3 Amazon
Amazon
Route 53
CloudFront
Dev DEV APP DEV
Dev Admin
Instance
DynamoDB

VPN
Internet TUNNEL
VPN
Gateway
Developers
Endpoint
&
Hook Internal
Instance

RESERVED INSTANCES

Now that we know where our code is going,

how is it getting there?

Ship that code!
• How are we going to deploy our code?
– File shipping:
• Just text files
• Binaries
– Package bundling:
• RPMs
• Tarballs
– As an AMI:
• Bundle one of the above into an AMI
• How fast do we need to do this? Yes, you can technically ship
• Across how many instances? your code to AWS in a box. See
Import/Export.
• How do we roll back (or forward)?

File Shipping Deploy Method
• Can be easier to work with than AMI method and package bundling
• Push out the code
– From Git/SVN/staging host
• Rolling restarts for web/application servers
• Leave existing hosts in place
• Have to worry about the cut over period
• Have to worry about feasibility of roll back/forward
• Can do deploy time schema changes (though a bad idea!)
• Have to worry about tracking what version is live for building new hosts

Deploying – Package Building
• Depending on the language/deployment method, you might need to take the
time to package your code.
– RPM
– Deb
– Something else?
• Throw this in as a step after a successful CI run.

Look at using tools like FPM to manage building packages for different
distributions.

AMI Deployment Method
• Code gets bundled into an AMI, we then deploy that AMI
– Pluses
• Very atomic
• New shouldn’t effect older versions
• Can deploy alongside current
• Easy tools to automate
– Cons
• Bit more work involved
• Have to think about where your data is persisting
• Schema updates potentially harder to package in
• Leverage configuration management tools in automation process

A quick aside - Schema updates
Schema changes tied to deployments are a huge blocker to moving fast.
– Hard to undo a change
– Can take a long time on SQL-based databases

Unlink this from code deploys:
– Flag on/off new features that touch the database in new ways
– Don’t make destructive database changes until no code touches that data
• No deletes, alters to live data! Ever!
– When altering existing data, opt instead to create a parallel column, copy data to new
column, then delete old
– Use “shadow queries” to test new functions/data sources for a percentage of users
before turning live to all

AMI Deployment Method - Building


Fully Functional
OS-Only AMI
AMI

Partially
Configured AMI


Fully Functional
OS-Only AMI
AMI

Least flexible
to maintain

Partially
Configured AMI


Fully Functional
OS-Only AMI
AMI

Least flexible Most amount of
to maintain post-boot work

Partially
Configured AMI


Fully Functional
OS-Only AMI
AMI

Least flexible Try and find a happy Most amount of
to maintain medium here post-boot work

Partially
Configured AMI

AMI Deployment Method - Deploying
Blue/Green Deploys Amazon
Route 53
– We stand up a duplicate part of our
100%
infrastructure and slowly cut traffic
over to it
• Shift via DNS ELB
• Makes it easy to do testing of new
features
• Makes it easy to roll back
– As we shift more traffic over, let auto-
scaling grow/shrink our instances of EC2 Instances
the new or old application
• Shut down the old when no traffic
there
MySQL RDS ElastiCache
DynamoDB
Instance Cache Node

Route 53
– We stand up a duplicate part of our 90% 10%
over to it
• Shift via DNS
ELB ELB
features
EC2 Instances EC2 Instances
scaling grow/shrink our instances of
there
DynamoDB
Instance Cache Node

Route 53
over to it
• Shift via DNS
ELB ELB
features
there
DynamoDB
Instance Cache Node

Route 53
– We stand up a duplicate part of our 100%
over to it
• Shift via DNS
ELB
features
scaling grow/shrink our instances of EC2 Instances

there
DynamoDB
Instance Cache Node


Netflix – Asgard
– Open Source tool
– Released in 2012
– “web-based tool for managing
cloud-based applications and
infrastructure.
– Helps do Blue/Green Deploys
– Capable of much more!

But how do we do all this quickly and
easily many times a day?

We need robots

Amazon
SWF

Automating the Process with Robots
Amazon Simple Workflow (SWF)
• Orchestration tool across your infrastructure Amazon SWF
• Use it as a middle layer to pass messages and setup tasks to be completed
• Break down individual tasks into different workers
• You define logic between workers
• Anything that can be scripted, can be made into a worker task
• Built in retries, timeouts, logging
• Low cost, reliability, and scalability built in

YOUR CODE = &
Deciders Workers

Automating the Process with Robots
Amazon Simple Workflow (SWF)
• Orchestration tool across your infrastructure Amazon SWF
• Use it as a middle layer to pass messages and setup tasks to be completed
• Break down individual tasks into different workers
• You define logic between workers
• Anything that can be scripted, can be made into a worker task
• Built in retries, timeouts, logging
• Low cost, reliability, and scalability built in

YOUR ROBOTS = &
Deciders Workers

Automating the Process
Workers: Workers
• Bundling code into an RPM – WORKER
• Making a new AMI with this RPM – WORKER
• Deploying a new CloudFormation stack with this RPM – WORKER
• Swapping DNS over to our new stack – WORKER
• Copy AMI across to another region for DR – WORKER
• Clean up old AMIs – WORKER

You get the picture.

Our Development &Test Infrastructure

3. Deploy RPM to Dev 53
Amazon S3
Amazon
Amazon
Decider CloudFront
Route
Dev
MySQL DB
Determines next step
DEV APP
ELB
DEV
WEB ELB
Dev Admin
Environment
Dev Stack Dev Stack Instance
DynamoDB

2. Build an RPM VPN Internet
VPN
TUNNEL
Gateway Developers
Endpoint
&
1. After CI Pre-commit
run
Hook
Amazon SWF
Operations
Internal
CI Workers
kicks off SWF
CI Server Git NAT
Instance
Execution
Amazon
SQS

Our code has arrived at its destination

Our code has arrived at its destination

But what now?

Monitoring/Logging Infrastructure

• Need to know what’s going on
• Spend the time required to do this well
• Share access to these tools with whole team
• Track every single resource that you can
• Alert on services, their availability, response times
• Make use of different cost models for different parts of this stack
• Try to keep log and other monitoring data for as long a possible
– 6 months? 1 year? Multiple years?

Monitoring/Logging Infrastructure
Tools:
• Logging
– Logstash
• Check out Kibana!
– Graylog2
– Syslog-ng/rsyslog/syslog
• Metrics
– CloudWatch
– Ganglia
– Graphite
• Monitoring
– Nagios
– Munin
– Sensu

HOST AGGREGATE
LEVEL LEVEL
METRICS METRICS

LOG ANALYSIS BUILD METRICS

AWS Marketplace can help
AWS Online Software Store
• Customer can find, research, buy software
• Simple pricing, aligns with EC2 usage model
• Launch in minutes
• Marketplace billing integrated into your AWS
account
• 600+ products across 23 categories

Developer Tool Categories Include
• Bug Tracking
• Monitoring
• Source Control
• Testing

Learn more at: http://aws.amazon.com/marketplace

Continuous Integration & Deployment on AWS

• Treat infrastructure as code.
• Automate the testing/deploy process end to end.
• Make sure environments mimic each other as closely as possible.
• Use repeatable patterns between environments at a different
scale.
• Use different cost models where it makes sense.
• Simplify and streamline the deploy process.
• Let AWS services handle control flows.
• Track everything (instance metrics, application metrics, logs).

Continuous Deployment Practices, with Production, Test and Development Environments Running on AWS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (18)

Ähnlich wie Continuous Deployment Practices, with Production, Test and Development Environments Running on AWS

Ähnlich wie Continuous Deployment Practices, with Production, Test and Development Environments Running on AWS (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Continuous Deployment Practices, with Production, Test and Development Environments Running on AWS