DevOps and Continuous Deployment @ WWPS Government, Education, and Non-profit Symposium 2014

AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
DevOps on AWS
Continuous Integration and Deployment Best Practices on AWS
Leo Zhadanovsky, Senior Solutions Architect, AWS | @leozh
JP Schneider, DevOps / Internet Jedi, Mozilla Foundation | @jdotp

Washington, DC | June 24, 2014 – June 26, 2014
Who Am I?
I work for AWS
I worked for the DNC 2009-2012
I was embedded in the
DevOps Team @ OFA
AWS does not endorse
political candidates

Who Am I?
JP, DevOps for Mozilla Foundation
Previous gig DevOps at OFA 2012
Before that, Ops at Threadless
@jdotp
Mozilla Foundation does
endorse animated cats

CONTINUOUS
INTEGRATION

What is Continuous Integration?
Changes to code automatically deployed to mainline
branch
• After passing unit and mock tests
Makes changes to code, and deployments iterative, not
monolithic
Bugs are detected quickly
Allows rapid development
Helps automate deployments

DEVELOPER

SOURCE CODE
REPOSITORY

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
PICK
TASKS

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
SUBMIT
CODE

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
SCHEDULE
BUILD

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
RECURRENT
BUILDS

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
CODE
FETCH

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
CODE QUALITY
TESTS
TEST
RESULTS

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
BUILD OUTPUT

SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
DOCS
BINARIES
& PACKAGES
DEV FACING
NOTIFICATIONS

SOURCE CODE
REPOSITORY
DNS
CONTINUOUS
INTEGRATION SERVER
PROJECT
MANAGEMENT SERVER
BUILDS

PAIN POINTS
• UNIT TESTS INCOMPLETE
• MOCK TESTS MAINTENANCE
• EXPENSIVE TEST ENVIRONMENT
• TEST ENVIRONMENT ≠ PRODUCTION
• DEPLOYMENT CYCLES

ON-DEMAND
PAY AS YOU GO
ELASTIC

=
PROGRAMMABLE
PLATFORM

IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT

A lot of options…
Configuration Management Systems
• Puppet
• Chef
• Saltstack
Deployment Frameworks
• Elastic Beanstalk
• OpsWorks
• Ansible
• Fabric
• Capistrano
Infrastructure Management
• CloudFormation

Bake an AMI Configure dynamically
Time consuming
configuration (startup time)
Static configurations (less
change management)
Bootstrapping

Continuous deployment
(latest code)
Environment specific
(dev-test-prod)
Bootstrapping
Bake an AMI Configure dynamically

Obama for America
awsofa.info

So here’s the Idea
~30th biggest E-commerce operation, globally
~200 distinct new applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users

a few constraints…
~30th biggest E-commerce operation, globally
~200 distinct applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users
Critically compressed budget
Less than a year to execute
Volunteer and near-volunteer development team
Core systems will be used for a single critical day
Constitutionally-mandated completion date

Web-Scale Applications

500k+ IOPS DB Systems

Services API

Business as usual..
…for a technology startup

Election Day – OFA Headquarters

Typical Charts

How?

The old approach, even from Amazon 

The old approach.. Might have some problems..

OFA’s Infrastructure
awsofa.info

Ingredients
Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic
EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3
ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire
nagios Paypal CentOS CloudSearch levelDB mongoDB python
securitygroups Usahidhi PostgresSQL Github apache bootstrap
SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS
Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch
ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS
Objective-C DynamoDB

Infrastructure, Configuration
Management & Monitoring
Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic
EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3
ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire
nagios Paypal CentOS CloudSearch levelDB mongoDB python
securitygroups Usahidhi PostgresSQL Github apache bootstrap
SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS
Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch
ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS
Objective-C DynamoDB

Configuration Management: Puppet
In mid-2011, we looked at options for configuration
management and chose Puppet
We needed to make it scale, and to get it to work with
state-less, horizontally scalable infrastructure
How did we do this?

Bootstrapping Puppet with CloudInit
CloudInit is built
into Ubuntu and
Amazon Linux
• Allows you to
pass bootstrap
parameters in
Amazon EC2
user-data field,
in YAML format

Don’t store creds in puppet manifests, store them in private
Amazon S3 buckets
Either pass Amazon S3 creds through CloudInit:
Even better – avoid this by using AWS Identity and Access
Management (IAM) roles and AWS Unified CLI’s S3 client

Built-in puppet support
Use certname with %i for instance id to name the node
Puppetmaster must have auto sign turned on
• Use security groups and/or NACLs for network-level security
In nodes.pp, use regex to match node names

Puppet Tips
Use a base class to define your standard install

Use runstages
Don’t store credentials in puppet, store them in private Amazon S3
buckets
• Use AWS IAM to secure the credentials bucket/folders within that bucket
Puppet Tips

Puppet Tips
Use puppet only for configuration files and what makes
your apps unique
For undifferentiated parts of apps, use Amazon S3
backed RPM/Debian repositories
• Can be either public or private repos, depending on your needs
• Amazon S3 Private RPM Repos: http://git.io/YAcsbg
• Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

Puppet Tips
By using packages for applications deploys, you can set
ensure => latest, and just bump the package in the repo
to update
Log everything with
rsyslog/graylog/loggly/NewRelic/splunk

Scaling the Puppet Masters
Use an Auto Scaling group for puppet masters
• Min size => 2, use multiple Availability Zones
Either have them build themselves off of existing puppet
masters in the group or off packages stored in Amazon
S3 and bootstrapped through user-data
Auto-sign must be on

One thing that is difficult to prepare for…

They had this built for the previous 3
months, all on the East Coast.

They had this built for the previous 3
months, all on the East Coast.
We built this part in
9 hours to be safe.
AWS +
Puppet +
Netflix Asgard +
WAN Optimization Software +
DevOps =
Cross-Continent Fault-
Tolerance On-Demand

If OFA was run on AWS today, what might that
look like?
Take advantage of OpsWorks
• This would simplify configuration management
Use CloudFormation to recreate environments for prod/dev/test quickly
Utilize PostgreSQL RDS instead of the many manually installed
PostgreSQL instances
Analytics data warehouse took months to build, RedShift would take
minutes
Docker containers could be used to easily allow developers to replicate
their dev environments in prod or staging
Use of immutable infrastructure
More use of CloudFront, resulting in less load on EC2 resources

Mozilla Foundation

Webmaker.org circa 2012
• Included Apps, non-SOA: Thimble, Popcorn, Goggles
• ~20 pushes of new software in 2012
• Operations and Development interacted mostly through bugzilla
tickets for deploys.
• Hosting in physical datacenter at Mozilla
Webmaker.org circa early 2013
• Deciding to go 12-factor, SOA in app layer
• Weekly pushes of Popcorn on train model
• Operations and Development interacted mostly through bugzilla
tickets for deploys.

April 2013
Webmaker begins rebuilding entire platform
SOA, 12-factor in node.js exclusively
Moving apps into AWS and DevOps / CI

Since April 2013….
Openbadges, Webmaker combine for: 1339 Pushes
Pushes Per Day to Staging / Prod
Pushes Per Day (Staging and Prod)

Who?
• ~30 Paid Developers
• Hundreds of Students
• Thousands of Contributors
• One DevOps / Internet Jedi
• Multiple Teams
How?
• Puppet, Jenkins, Fabric
• Tight feedback loops:
Newrelic, Opsview
• Culture Shift
• Staging Envs
• Brave devs iterate,
keeping work in-context
• Visible Ops
• Cross-train developers in
operations

What changed?

1) Know How You Were Doing Before
2) Know What Changed When
3) Know How You Are Now Doing
=
The confidence to try more things
and try them faster, with minimum
viable planning.

Deployment
Pipeline

AWS
CLOUDFORMATION
STACK-BASED DEPLOYMENT
SERVICE

{
"Description" : "Create RDS with username and password",
"Resources" : {
"MyDB" : {
"Type" : "AWS::RDS::DBInstance",
"Properties" : {
"AllocatedStorage" : "500",
"DBInstanceClass" : "db.m1.small",
"Engine" : "MySQL",
"EngineVersion" : "5.5",
"MasterUsername" : "MyName",
"MasterUserPassword" : "MyPassword"
}
}
}
}

"AWS::CloudFormation::Init" : { "config" : {
"packages" : {
"yum" : {
"mysql" : [],
"mysql-server" : [],
"httpd" : [],
"php" : [],
"php-mysql" : []
}
},
"sources" : {
"/var/www/html" :
"https://s3.amazonaws.com/my-builds/build-v4.zip"
}
}

{
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2
KeyPair to enable SSH
access to the instance",
"Type" : "String"
}
},
}

CLOUDFORMATION
TEMPLATE
PROCEDURAL
DEFINITION
Create it programmatically
KNOWN
CONFIGURATION
Store stack configuration in
source control
PARAMETER
DRIVEN
Dynamic and user-driven
templates
COLLABORATION
Share templates with ease
as just files

APPLICATION
VERSIONS
+
INFRASTRUCTURE
VERSIONS

NASA/Jet Propulsion Laboratory
California Institute of Technology

Mars Rover Landing by the #’s
NASA TV = HD stream, 1080p, ~1 mb/s per viewer
Expecting peak of ~1m viewers
All playback devices (iOS, Android, Flash, HTML5, blah)
Once in a lifetime moment in history (no crashy crashy)
FUN FUN FUN

Mars Rover Landing by the #’s
NASA TV = HD stream, 1080p, ~1 mb/s per viewer
Expecting peak of ~1m viewers
All playback devices (iOS, Android, Flash, HTML5, blah)
Once in a lifetime moment in history (no crashy crashy)
NASA says we can’t use their live stream setup
It’s 6 days before the landing
It’s the same week as the Olympics
Available technical resources from JPL: Brett and Khawaja

The sticky wicket:
HD Video stream
size = 4mb
never changes
easy to cache
manifest.f4m
size = 4kb
New every 4 sec
caching difficult

The plan
Design a solution around our limits
• Max connections to origin = 6
• Max streams per cache node = 20
• Local Latency = critical
• US-WEST-1 capacity reserved for S3 static
images
Test the snot out of it
Hang on!

ok, so one of them is a rocket scientist..

Mars Science Laboratory - Live Video Streaming
“The 42 pack”

LOAD TESTING

Battle Testing our Deployment
Benchmarking

Dynamic Traffic Scaling
US-East Cache Node Performance 25.3 Gbps

Only ~42Mbps
Dynamic Traffic Scaling
Impact on US-East FMS Origin Servers

CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES
CONSTANTLY INTEGRATING INTO
PRODUCTION.

KEY = ITERATION

ITERATION
=
MODIFY THE SYSTEM TO BETTER
MEET THE EXPECTATIONS OF
YOUR USERS

11.6s
Mean time
between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean number of
hosts
simultaneously
receiving a
deployment
30,000
Max number of
hosts
simultaneously
receiving a
deployment
DEPLOYMENTS AT
AMAZON.COM

SOFTWARE DEPLOY
≠
PRODUCT LAUNCH

1.5 BILLION PAGE VIEWS
OCTOBER 2012
$83 MILLION IN TRANSACTIONS
4.2 MILLION ITEMS SOLD

30 DEPLOYS PER DAY
1 DEPLOY EVERY 20 MINUTES

"Production is truly the only place you
can validate your code."

AWS OPSWORKS
INTEGRATED APPLICATION
MANAGEMENT
AWS OPSWORKS UNDER THE HOOD

DATA-DRIVEN
ARCHITECTURES

METRICS @ETSY

METRICS @OBAMA FOR AMERICA

COST-ORIENTED
ARCHITECTURES

PHP+APACHE+VARNISH
NGINX+NODEJS

CONTINUOUS
INTEGRATION
CONTINUOUS
DEPLOYMENT

CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
EXPERIMENTATION

CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
IMPROVEMENT

INNOVATE

« Want to increase innovation?
Lower the cost of failure »
Joi Ito

SPEED AND AGILITY
Experiment
Often
Fail quickly at
a low cost
More
Innovation
Experiment
Infrequently
Failure is
expensive
Less
Innovation
“ON-PREMISE”

Q & A

aws.amazon.com/training
Expand your technical
expertise to design,
deploy, and operate
scalable, efficient
applications on AWS
Training
aws.amazon.com/certification
Certification
20% off AWS Instructor-Led Training Class*
• Code MK20PSSYM314
• Expires September 30, 2014
Get Savings on AWS Training!
* Discount only applies to classes delivered by AWS. Discount
does not apply to classes delivered by an APN Training Partner.
aws.amazon.com/training/self-paced-labs
Self-
Paced
Labs
Get hands-on
practice working with
AWS technologies in
a live environment
Validate your proven
technical expertise
with the AWS
platform
Expand your skills with AWS

DevOps on AWS
Continuous Integration and Deployment Best Practices on AWS
Thank You
Leo Zhadanovsky, Senior Solutions Architect, AWS | @leozh
JP Schneider, DevOps / Internet Jedi, Mozilla Foundation | @jdotp

DevOps and Continuous Deployment @ WWPS Government, Education, and Non-profit Symposium 2014

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (9)

Ähnlich wie DevOps and Continuous Deployment @ WWPS Government, Education, and Non-profit Symposium 2014

Ähnlich wie DevOps and Continuous Deployment @ WWPS Government, Education, and Non-profit Symposium 2014 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

DevOps and Continuous Deployment @ WWPS Government, Education, and Non-profit Symposium 2014

Hinweis der Redaktion