With AWS companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100% API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session we'll talk about some key concepts and design patterns for Continuous Deployment and Continuous Integration, two elements of lean development of applications and infrastructures.

1. AWS Government, Education, &
Nonprofits Symposium
Canberra, Australia | May 20, 2014
Continuous Integration and Deployment
Best Practices on AWS
Adrian White
Solutions Architect,
Amazon Web Services

2. Innovation, Quality, Governance

3. Stacks /
Environment(s)
An example CI / CD workflow
CI/CD tool
Issue
Tracker
SCM
Infrastructure
automation /
deployment
Test tools /
harnesses

4. CloudFormation Stack(s)
An example CI / CD workflow
PHPUnit
jQuery
…
Tasks for AWS

5. A CI / CD pipeline
Teardown
Release
Deploy
Test
Code

6. Get your source under control
Prod Phoenix (feature)UAT
Code
Test
Deploy
Release
Teardown
Code
Test
Deploy
Release
Teardown
Code
Test
Deploy
Release
Teardown
Code
Test
Deploy
Release
Teardown

7. Automated Merging
UAT Feature 2Feature 1

8. What does CI give us?
• Test driven promotion (of development change)
• Increasing velocity of feedback cycle through
iterative change
• Contain change to reduce risk
• Bugs are detected quickly
• Automated testing reduces size of testing effort

9. What does CD give us?
• Changes are pushed quickly to production
• Immediate feedback from users
• Supports A/B testing or “We test customer
reactions to features in production”
• Hardens, de-risks the deployment process
• Gives us a breadth of data points across our
applications

10. Deployment approaches
• Deploy in place
– Manage interruption
• Bake
– Discrete environment
• Multiple environments from branches
• Support A/B testing
• “Rolling DNS”
Deploy Deploy in-place
Bake

11. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

12. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

13. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

14. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

15. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

16. Deploy in-place
Un-baked
• Common baseline AMI
• Customise at instantiation
• Change in-place
Your app
AMI

17. Bake process
1. Start a builder instance
2. Bootstrap / cfn-init, cfn-signal
3. Bake your AMI
4. Tag it
5. Destroy/clean up the builder instance

18. cfn-init
"AWS::CloudFormation::Init" : {
“cfn” : {
"packages" :{
"yum" : { "httpd" : [] }
},
"files":{
“/home/ec2-user/myfile.html:{
"source" : { "Fn::Join" : [ "", ["https://s3-ap-
southeast-2.amazonaws.com/",{ "Ref" : "S3Bucket" },”/myfile.html”]] },
"mode":"000644",
"owner":"root",
"group":"root",
"authentication":"S3AccessCreds”
},

19. cfn-init
"services": {
"sysvinit" : {
"httpd" : {
"enabled" : "true",
"ensureRunning" : "true"
}
}
}
}

20. Release
Release
Test the new stack
Match the traffic between the two stacks
Update the “floating” DNS record
Send Notification(s)
ROLLBACK

21. Blue green deployment
awssummit-1.com awssummit-2.com
awssummit.com

22. Blue green deployment
awssummit-1.com awssummit-2.com
awssummit.com

23. Blue green deployment
awssummit-1.com awssummit-2.com
awssummit.com

24. Blue green deployment
awssummit-1.com awssummit-2.com
awssummit.com

25. Teardown
Teardown
Ensure that no traffic is moving though ELB
Teardown the CloudFormation Stack
Deregister the AMI

26. Advanced Techniques
• Extending CloudFormation with custom
resources
• Managing CD sprawl
• Segregation of duties
• Extending your CD tools

27. Stack chaining

28. Stack chaining

29. Stack chaining

30. CloudFormation Merging
Git Git
CloudFormation
Operations Repo Application Repo
VPC
Subnets
Security Groups
CloudFormation Frameworks
Best Practice
Application Code
Application CloudFormation
Load Balancing Setup

31. CloudFormation Custom Resources
• Change DB schema during deployment
• Extend CloudFormation to support other
services - “So You Think You Are An
AWS Ninja” talk
https://github.com/aws/aws-cfn-resource-bridge
https://github.com/awslabs/aws-cfn-custom-resource-examples
Parameters
Custom resource
implementation
Git

32. Custom resources – DatabaseSchema
"MyDBSchema" : {
"Type" : "Custom::DatabaseSchema”, "Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:12345EXAMPLE:DBSchema",
"databaseChangeLog" : [ {
"changeSet" : {
"id" : "1",
"author" : "adamthom",
"changes" : [ {
"createTable" : { … }
} ]
}
}
}
}

33. Custom resources – DatabaseSchema
"createTable" : {
"tableName" : "example",
"columns" : [ {
"column" : {
"name" : "id",
"type" : "int",
"autoIncrement" : true,
"constraints" : {
"primaryKey" : true,
"nullable" : false
}
}
} ]
}

34. Extending your CD tools
Tasks for AWS
DynamoDB

35. Situational Awareness
Burden of Responsibility
APIs
Tasks for AWS

36. Containerisation
• Build environments for artifacts, don’t update
environments with artifacts
• All environments are transient
• Standardisation, abstraction and portability

37. Docker, Amazon Linux and Elastic
Beanstalk
• A framework for managing containers
• LXC containers are more lightweight than VMs
• Amazon Linux (2014.03) bundles Docker 0.9
and LXC 0.9
• Docker containers on Beanstalk are Go!

38. Innovation, Quality, Governance
Discrete environments for each branch
Automated testing on every commit on every branch
Leverage CD tools to provide separation of duties
Audit Logs
Git approvals process
Use custom resources to extend CloudFormation
Leverage DNS
Interface with the API
Environments for artifacts

39. Customer: Open Training Institute

40. Journey to the cloud
Steve Mactaggart
Manager, Open Training Platform

41. • Our
core
Higher
Educa/on
business
– Open
Universi/es
Australia
is
a
na/onal
leader
in
online
higher
educa/on.
– Through
Open
Universi/es
Australia
you
can
study
high-­‐
quality
degrees
with
leading
Australian
universi/es.
Introduction to Open Universities

42. • Launch
of
Open2Study
– Develop
Australia’s
first
large
scale
MOOC
– Enhance
our
capability
to
design
and
deliver
online
learning
– New
technical
plaIorm
for
us
to
deliver
• Now
Open
Training
Ins/tute
– A
new
business
focused
on
delivering
Voca/onal
Educa/on
and
Training
online
– Leveraging
our
experience
of
the
exis/ng
Open
business,
capabili/es
developed
for
Open2Study
– Project
kickoff
May
2013,
Launch
Dec
2013
Our Journey

43. • New
technology
stack
• Ambi/ous
scope
of
work
• Lots
of
compe/ng
business
objec/ves
• Aggressive
/meframes
• Unexpected
ini/al
load
and
growth
profile
• Geographically
separate
development
team
Challenges

44. What does Open2Study look like today
170,000+
students
300,000+
enrolments
4,700,000+
min
of
video
From
over
200
countries
Web
instances
scale
between
4
-­‐-­‐>
12
m1.xlarge
instances

45. • Stage
1:
Spike
it
up
(POC)
– AWS
CLI
build
AS
group
and
launch
• Stage
2:
Automated
deployments
– Ini/al
version
of
deployment
script,
manually
triggered
by
logging
on
to
server
• Stage
3:
Deployment
pipeline
– Jenkins,
ssh
and
Build
pipeline
plugin
• Stage
4:
Prepare
for
scale
– Manual
AMI
crea/on
and
rollout,
Using
AWS
CLI
to
reconfigure
the
ASG
Our path to successful scale

46. • Lessons
from
Ini/al
spike
– AWS
for
prototyping
was
really
painless
– AMI
provisioning
managed
by
AS
group
– Scale
Up/Down
rules
using
CloudWatch
– CPU
rules
caused
early
scale
up
– Need
for
becer
metrics
to
trigger
scale
– Regular
adjustment
required
• Lessons
from
Automated
deployments
– Reproducible
steps
with
basic
script
– Script
needs
to
be
version
controlled
– Deployment
from
git
– AWS
looks
and
feels
very
similar
to
our
exis/ng
infrastructure
Key learning’s

47. • Lessons
from
Con/nuous
delivery
– Manual
tasks
in
pipeline
for
approval
– Prime
the
pipeline
on
every
developer
change
– Constantly
extend
‘deploy’
script
– BASH
is
a
good
start,
but
not
the
end
game
• Lessons
from
prepare
for
scale
– Manual
AMI
crea/on
soon
became
blockage
– Convert
process
to
simple
BASH
script
– Integrate
with
deployment
pipeline
– ASG
to
replace
applica/on
Key learning’s

48. • Enter
OpsWorks
– Provides
server
configura/on
management
– Lifecycle
events
to
manage
deploy
process
– Automated
Load
and
Time
based
scaling
• Lessons
from
Configura/on
management
– New
set
of
skills
required,
chef
cookbooks
– Re-­‐architect
the
development
and
deployment
process
– Scale
up
slow
(~15min
to
provision
node)
– Python
/
boto
for
deployment
automa/on
control
Where we are today

49. • Repeatable
ar/facts
promoted
between
environment
– Solu/on
is
somewhere
between
baked
AMI
'fixed
baked'
approach
and
'build
on
demand'
process
of
OpsWorks
• Standard
ar/facts
with
configura/on
injec/on
– Configura/on
injected
via
user
data
– Read
and
configured
from
S3
• Full
automa/on
of
all
applica/on
and
environment
changes
– Flexibility
of
build
from
scratch
Where we see we are heading

50. THANK YOU
Please give us your feedback by filling out the Feedback Forms
AWS Government, Education, &
Nonprofits Symposium
Canberra, Australia | May 20, 2014