As a product grows, and the infrastructure becomes more complex, the Operations team traditionally shoulders the burden of maintaining this infrastructure while deploying code from Software Engineers. Code is sometimes given to Operations with little to no information regarding how it should run or what the criteria for successful deployment is. This is not due to lack of caring, Software Engineers often lack the context themselves to provide production deployment instructions. To Software Engineers, production can be like a walled off city, filled with pathways and rooms not to be explored, guarded by Operations. This presentation aims to provide a solution to this problem. We will address how the traditional separation of Operations and Software Engineers slows innovation, and redefine their relationship -- blending responsibilities. We will examine the transition of two real teams, an Operations team and Engineering team, from complete isolation, to closer environments through virtual machines, to one cloud environment shared by all and managed with CloudFormation.

1. James Andrew Vaughn (Andy) @MindTouch
Tear It Down, Build It Back Up:
Empowering Developers with
Amazon CloudFormation

2. James Andrew Vaughn (Andy)
• Software Architect at MindTouch
• @modethirteen on Twitter & GitHub
• Interests
• Software Build and Testing Automation
• Frontend Web Performance
• Web Components & Polymer
• SSO and Identity Management

3. @modethirteen
Agenda
• What is Amazon CloudFormation? Why use it?
• Managing your release testing and production infrastructure
code
• Give developers the power (`cause knowledge is power!)

4. @modethirteen
Why manage infrastructure
as code?

5. @modethirteen

6. @modethirteen
All of our customers host their brand on our
common, hosted infrastructure. One mistake
and all customer brands look bad #yousuck

7. @modethirteen
Before CloudFormation
• Infrastructure had grown organically over years
• Hand rolled scripts with boto.py to create different EC2 instance
types, and manual Puppet runs to configure them
• Non EC2 AWS Resources managed by hand
• No infrastructure in different zones or fast, programatic disaster
recovery for entire infrastructure
• Developers were ignorant of production infrastructure

8. @modethirteen
Weekly releases must be simple,
repeatable, non events

9. @modethirteen
Developers cannot be isolated from
the infrastructure where their code
will ultimately run

10. @modethirteen
Code gives context to problems
solved and provides audit trail for
infrastructure design

11. @modethirteen
Infrastructure code and server
configuration code is versioned with
application code

12. @modethirteen
CloudFormation: Define creation of AWS
resources (EC2 as well as Security
Groups, SQS, RDS, etc)
Puppet, Chef, SaltStack, Ansible: Define
actions that occur within EC2 instances
once they’ve been provisioned

13. @modethirteen
CloudFormation vs Terraform
• Access to nearly every AWS
resource. Better support for
VPC, Security Groups, IAM,
Cloudfront, SQS
• Stable and mature
• JSON infrastructure templates
can be generated by
Troposphere (with Python
logic)
• Vendor neutrality: AWS, OpenStack, Heroku,
etc
• Can execute infrastructure plans as a dry run
• DSL for generating infrastructure templates
(HCL)
• If one resource fails to build, subsequent
rebuild will only build tainted resource and
those dependent on it
• Open source so AWS API coverage can be
improved by community
Google Docs: Terraform AWS Coverage

14. @modethirteen
CloudFormation Stacks
Main Stack
Sub Stacks
A stack is a collection of AWS resources that can be configured

15. @modethirteen
App Server Pool
Stack
Database
Stack
ElasticSearch
Stack
App Server Pool
Stack
Main Stack

16. @modethirteen
CloudFormation Stacks
Resources are things that can be queried, configured in the AWS API (including CloudFormation
sub stacks). Examples: Listing S3 buckets, Adding Route 53 DNS entries, Taking DB snapshots

17. @modethirteen
Database
Stack
ElasticSearch
Stack
App Server Pool
Stack
Main Stack
• AutoScaling::AutoScalingGroup
• AutoScaling::LaunchConfiguration
• IAM::InstanceProfile
• IAM::User
• AutoScaling::AutoScalingGroup
• AutoScaling::LaunchConfiguration
• CloudFormation::WaitCondition
• IAM::InstanceProfile
• IAM::User
• RDS::DBInstance
• IAM::InstanceProfile
• IAM::User

18. @modethirteen
Custom Resources
• CloudFormation::CustomResource
• Sends custom HTTP message (Service Token) to any of your endpoints, and
continues stack execution after response
• AWS SNS
• AWS Lambda
• Node.JS
• Your choice!

19. @modethirteen
CloudFormation Stacks
Stack parameters come from API input, version controlled JSON templates,
or from the output of other stacks

20. @modethirteen
• MySQL Storage Engine
App Server Pool
Stack
Database
Stack
ElasticSearch
Stack
App Server Pool
Stack
Main Stack
• ElasticSearch Version
• App Server Pool EC2 Group Name
• ElasticSearch EC2 Group Name
• RDS MySQL IP & Port

21. @modethirteen
CloudFormation Stacks
Parameters of stack can be outputted to dependent stacks. Example: IP’s,
Security Policies, Custom Values, etc.

22. @modethirteen
Template: {…}
App Server Pool
Stack
Database
Stack
ElasticSearch
Stack
App Server Pool
Stack
Main Stack
• MySQL Storage Engine
• ElasticSearch EC2 Group Name
• RDS MySQL IP & Port
• ElasticSearch Version
• App Server Pool EC2 Group Name
Template: {…}

23. @modethirteen
Stack Policy: Stack Update
Resource Access Control

24. @modethirteen
Deploying a Stack

25. @modethirteen
Troposphere

26. @modethirteen
Puppet / Chef / SaltStack / Ansible
• Stack includes an EC2 Instance or AutoScaling Group Resource
• Resource includes a “UserData” metadata section, for bootstrapping an instance or group of
instances
• Include data that cloud-init uses to install instance configuration tool of choice
• curl http://169.254.169.254/latest/user-data
• Example:
• cloud-init installs puppet from UserData commands
• cloud-init runs puppet (configures instance and installs cfn-signal)
• cfn-signal notifies CloudFormation that puppet was success or failure

27. @modethirteen
Execute Deployment

28. @modethirteen
Lessons Learned
• Goal was to put entire existing AWS infrastructure into CloudFormation, no
immediate value was attained
• Difficult getting buy in for incremental improvements to infrastructure
management
• Existing resources cannot be migrated to CloudFormation
• Know the caveats of deleting AWS Resources, they can fail a stack tear down
• AWS Resources missing from CloudFormation API can be mitigated with
Custom Resources
• Must understand what a resource does when it updates

29. @modethirteen
Send in the Developers

31. @modethirteen
Approach #1 : Build your own web
console for launching test
and dev stacks

32. @modethirteen
Approach #2 : Every developer has
their own AWS account billed to main
AWS account

33. @modethirteen
Approach #3 : One developer AWS
account billed to main account

34. @modethirteen
The Teams
• Are developer teams responsible for their own container /
infrastructure templates, are operators part of these teams
• Are developers just as responsible for troubleshooting when
infrastructure goes down
• What are operator obligations to developers
• What are developer obligations to operations

35. @modethirteen
TL;DR
• Your product is application code, data, services, and servers
• CloudFormation deploys your product to production
• CloudFormation deploys your product for development and testing
• Your developers can make better decisions
• Your operators can make better decisions
• Your customers / users are happy

36. The End. Q?