An overview presentation given to the AWSNE Meetup on 2014-07-03. Covers the basics of combining EC2, SQS, CloudFormation and IAM to make a reusable worker tier.
4. EC2
Resizable compute capacity designed for
developers
Full access. Build what you need / Own what you
build
EC2 compute unit - consistent measure of instance
performance
Choice of operating systems
Tags
5. Amazon Linux
Supported and maintained by AWS
Lightweight and designed for EC2
Twice yearly releases - continual package updates
Compatible with EPEL
6. Unified CLI Tools
Unified interface to all* AWS products
Autocomplete and built-in help
Installed by default to Amazon Linux AMI
JSON
Supports EC2 IAM roles
13. The challenge
A scalable, fault tolerant worker tier that gets
messages from SQS and processes the output to
S3
Built through code and configuration without
manual intervention
Easy to deploy and maintain
14. CloudFormation - Elements
An optional list of template parameters (input
values supplied at stack creation time)
An optional list of output values (e.g. the complete
URL to a web application)
An optional list of data tables used to lookup static
configuration values (e.g., AMI names)
The list of AWS resources and their configuration
values
15. AWS Resources
An SQS queue to pull from
An autoscaling group of EC2 worker nodes
CloudInit config to configure the node
An IAM role and instance profile for the node to
assume
An S3 bucket to store output
A CloudWatch alarm to trigger scaling
Infrastructure as a service is about flexibility. Use it instead of PAAS. It’s a set of very well defined building blocks, it’s not a blank piece of paper. Sample architecture is available.
We’ll see how lots of tools available to help you build something robust
Location aware, can launch in multiple regions with parameter
Can use to kickstart chef / ansible / puppet etc - not a replacement
Worth the extra effort to have version-able reusable documented
> ElasticBeanstalk < Opsworks / Use for all three
Templates are monolithic but stacks (resources) can be combined via inputs/outputs
Usually paired with something to automate and link the calls.
Tags available on most things in amazon, but EC2 is where it benefits
Third-party tools
Better than hostnames, tag early, tag often.
Use tags to query values for the running instances, make decisions. We’ll see this later - good way to parameterise instances
RedHat certified engineer.
CentOS user since v4
Transitioning to Amazon Linux - heartbleed was patched and available within hours, general setup is excellent, best practices implemented
Learn to love them less, but, you still must love them
Xen HVM kernel available - easiest way to get enhanced hardware support for new instances
Lots of documentation around, version 1.3 of the tools is what I’m talking about. Python based.
On github.
Pair with JQ to make parsing fun
Very easy to use, self-contained within template. No need for external resources. Security handled through IAM
not restricted by user-data size limit 16KB, can take 512KB of data.
downsides is stored in template :(
most actions are covered with these basic commands
Just use IAM - naming is hard but worthwhile
Free to use
conditions - including source ip, time of day, force ssl etc
Security Assertion Markup Language - XML
Temporary credentials for ec2 instances delivered automagically via IAM roles
Don’t have to build everything yourself - mix and match
S3 is the daddy. Storage / Speed CDN / Permission model - very granular
SPA JS apps - integration with CORS even API SDK support for some services