Jinesh varia - AWS CloudFormation Puppet at PuppetConf

1. Architecting for the Cloud: AWS CloudFormation and Puppet Jinesh Varia

2. Scale, Pace of Innovation, Expansion and Ecosystem

3. Cloud Customers in 190 Countries

5. Boot from Amazon EBS» Amazon CloudFront Streaming » Amazon VPC enters Unlimited Beta » AWS Region in Northern California » International Support for AWS Import/Export » AWS Multi-Factor Authentication » Virtual Private Cloud » Lower Reserved Instance Pricing » Reserved Instances in EU Region » Elastic MapReduce » SQS in EU Region » Amazon RDS » High-Memory Instances » Lower EC2 Pricing » New SimpleDB Features » FPS General Availability » Amazon SNS » AWS Security Center 2009 Jan 2010 Jan Jul Sep Oct Dec Aug Nov Feb Mar Apr Jun May Feb Mar » Amazon EC2 with Windows » Amazon EC2 in EU Region » AWS Toolkit for Eclipse » Amazon EC2 Reserved Instances » Amazon CloudFront Private Content » SAS70 Type II Audit » AWS SDK for .NET » Amazon Elastic MapReduce in Europe » Amazon EC2 Reserved Instances with Windows, Extra Large High Memory Instances » Amazon S3 Versioning Feature » Consolidated Billing for AWS » Lower pricing for Outbound Data Transfer » AWS Import/Export » New CloudFront Feature » Monitoring, Auto Scaling & Elastic Load Balancing » EBS Shared Snapshots » SimpleDB in EU Region » Monitoring, Auto Scaling & Elastic Load Balancing in EU » Lower pricing tiers for Amazon CloudFront » AWS Management Console The pace of innovation in 2009

6. » Free Monitoring EC2 » Amazon Route 53 » PCI DSS Level 1 Certification » Mobile SDKs (Android, iPhone) » Large Object S3 Support » Florida POP » Import/Export APAC » Amazon SNS » Combined AWS Data Transfer Savings » Amazon EMR Bootstrap Actions » Amazon ELB Session Stickiness » Amazon RDS in EU » New Singapore Region » RDS Reserved » CloudFront Default Root » Startup Challenge 2010 » CloudFront Invalidation » AWS Elastic Beanstalk » Amazon Simple Email Service » Improved AWS Support “Bronze” » Amazon CloudWatch Console » CloudFront HTTPS » NYC Edge Location » Lowers Pricing HTTP » AWS Import Export GA » Amazon SNS » Amazon S3 Console » Amazon EBS CloudWatch » Amazon RDS Read Replicas » Suse EC2 Linux » Amazon SNS Console » Amazon ELB HTTPS » AWS Free Tier » EMR Resizing Cluster » EMR JobFlow Debugging » Simple DB Consistent Reads » Simple DB Conditional Puts » VM Connector » Tokyo Region » AWS Support JP 2010 Jan 2011 Jan Jul Sep Oct Dec Aug Nov Feb Mar Apr Jun May Feb Mar » New VPC » Dedicated Instances » Windows 2008 R2 » Amazon S3 Lowered Pricing » CloudFront GA, SLA » S3 Multipart » GPGPU Instance Types » ISO27001/2 Certification » Amazon SQS Longer retention, Free Tier Amazon S3 Bucket Policies » Amazon VPC IP Address » Cluster Compute Instances » Amazon S3 RRS Notifications » AWS Java SDK » Windows BYOL » Singapore Pop » CloudFront Private Streaming » Lowered Pricing EC2 » AWS IAM » Amazon VPC Console » Micro Instances » Amazon Linux AMI » Amazon EC2 Tagging, Filtering, Idempotency, » Oracle Certified AWS » AWS PHP SDK » AWS CloudFormation » Amazon S3 Static Websites » AWS IAM Website Login » Paris Edge Location » Amazon EC2 Reserved Instances with Windows, Extra Large High Memory Instances » Amazon S3 Versioning Feature » Consolidated Billing for AWS » Lower pricing for Outbound Data Transfer » VPC in EU » Amazon RDS in US-west » Amazon CloudFront Access Logs » Amazon RDS Multi-AZ » Amazon S3 RRS » Amazon RDS Console And pace accelerates in 2010….

7. “Every day is a launch day” » On-demand Red Hat » Stockholm Edge Location » AWS Elastic Beanstalk new enhancements » New Data Transfer pricing » Free Inbound Data Transfer » Spot Integration with HPC instances » Amazon EMR in APAC » AWS Mobile SDKs » Live Streaming with CloudFront » AWS IAM GA » AWS IAM Web Console » AWS Import/Export for EBS » AWS CloudFormation new features » AWS SDK for Ruby » Attachment support for Amazon SES » AWS Startup Challenge goes global » AWS DirectConnect » Amazon VPC Everywhere » Mulit-AZ VPC » AWS IAM Identity Federation » AWS toolkit of eclipse 2.0 » AWS GovCloud US » Spot in Amazon EMR » Amazon ElastiCache » Amazon VM import Win2k3 » VM Connector » Tokyo Region » AWS Support JP » AWS IAM for CloudFront » VPC Virtual Networking » VPC Internet Access 2011 Jan 2012 Jan Jul Sep Oct Dec Aug Nov Feb Mar Apr Jun May Feb Mar » AWS CloudFormation » Amazon S3 Static Websites » AWS IAM Website Login » Paris Edge Location » Amazon Route53 » New VPC » Dedicated Instances in VPC » Windows 2008 R2 » New AZ in JP » AWS IAM GA » AWS IAM Web Console » AWS Beanstalk Tomcat 7 Support » Amazon CloudWatch Custom Metrics » Amazon CloudWatch lower pricing » AWS SAP Certification » Amazon RDS for Oracle » Amazon ELB ipv6 support, Zone Apex » Amazon ELB Security Group integration » Amazon Route53 GA, ELB integration » Amazon Route 53 Weighted RR » New pricing control for Spot » AWS CloudFormation new enhancements » AWS Mobile SDK GA » AWS Toolkit of Visual Studio » AWS DirectConnect USWest Location » AWS Elastic Beanstalk » Amazon Simple Email Service » Improved AWS Support “Bronze” » Amazon CloudWatch Console

8. Each day, AWS adds the equivalent server capacity to power Amazon when it was a global, $2.76B enterprise (circa 2000)

9. GovCloud-US US West (Northern California) US East (Northern Virginia) Europe West (Dublin) Asia Pacific Region (Singapore) Asia Pacific Region (Japan) Ashburn, Dallas, Los Angeles, Miami, Newark, Palo Alto, Seattle, St. Louis, Amsterdam, Dublin, Frankfurt, London, Hong Kong, Singapore, Tokyo, New York, Paris Amazon CloudFront Edge Locations

10. The AWS Cloud Your Application Tools to access services Libraries and SDKs .NET/Java etc. Web Interface Management Console Tools AWS Toolkit Eclipse, VS Command Line Interface Cross Service features Auth, Authorization, FederationAWS IAM, MFA Monitoring Amazon CloudWatch Deployment and Automation AWS Elastic BeanstalkAWS CloudFormation High-level building blocks Content Delivery Amazon CloudFront Email Amazon SES Payments Amazon DevPay Amazon FPS Parallel Processing Amazon Elastic MapReduce Messaging Amazon SNS Amazon SQS Workforce Amazon Mechanical Turk Low-level building blocks Compute Amazon EC2 Auto Scaling Network Amazon VPC, ELB, DirectConnect Amazon Route 53 Storage Amazon S3 Amazon EBS Database Amazon RDS Amazon SimpleDB Amazon ElastiCache Amazon Global Physical Infrastructure (Geographical Regions, Availability Zones, Edge Locations)

11. www.yourApp.com media.yourApp.com (Static data) Amazon CloudFront Amazon Route 53 Elastic Load Balancer Amazon CloudWatch Amazon S3 Bucket Amazon SNS Notifications Auto Scaling Group Amazon SimpleDB App Tier Email ElastiCache Tier Amazon RDS AZ-1 AZ-1 Region

12. Corporate data center Availability Zone 1 DirectConnect Location 10G VPC Subnet Router VPN Gateway Customer Gateway Corporate Headquarters VPC Subnet Internet Gateway Amazon VPC Availability Zone 2 Branch Offices The New Cloud-Ready Enterprise IT Amazon S3 Amazon SES Amazon SimpleDB Amazon SQS AWS Region

13. The “Living” AWS Cloud Your Application Tools to access services Libraries and SDKs .NET/Java etc. Web Interface Management Console Tools AWS Toolkit Eclipse, VS Command Line Interface Cross Service features Auth, Authorization, FederationAWS IAM, MFA Monitoring Amazon CloudWatch Deployment and Automation AWS Elastic BeanstalkAWS CloudFormation High-level building blocks Content Delivery Amazon CloudFront Email Amazon SES Payments Amazon DevPay Amazon FPS Parallel Processing Amazon Elastic MapReduce Messaging Amazon SNS Amazon SQS Workforce Amazon Mechanical Turk Low-level building blocks Compute Amazon EC2 Auto Scaling Network Amazon VPC Elastic LB Amazon Route 53 Storage Amazon S3 Amazon EBS Database Amazon RDS Amazon SimpleDB Amazon ElastiCache Amazon Global Physical Infrastructure (Geographical Regions, Availability Zones, Edge Locations)

14. The Need for Speed

15. DevOps

16. Ops = businessOps != businessOps ? business

17. Ops Ξ business

18. Elasticity is the fundamental property of the cloud

19. Implement Elasticity Elasticity during the day 25% Savings

20. Implement Elasticity Elasticity during the year 50% Savings

21. Implement Elasticity Optimize during the month 75% Savings

22. Mr. Automate Development Automate Using Cloud APIs Management Logistics Monitoring Deployment

23. The Automation You Always Meant to Build Provision and attach 1TB of storage in 2 minutes (from the back of an auto-rickshaw in India). 10 new Linux servers in 2 minutes (while sitting by the pool on a nice day). Monitoring server resources from an iPhone (in a bar). Source: Autodesk

24. AWS CloudFormation“Provision your infrastructure stack using one script”

25. www.yourApp.com media.yourApp.com (Static data) Input Parameters Resources Outputs JSON Plain Text Perfect for Version Control Validate-able Mappings Custom Metadata Amazon CloudFront Amazon Route 53 Elastic Load Balancer Amazon CloudWatch JSON Template AWS CloudFormation Service Amazon S3 Bucket Amazon SNS Notifications Auto Scaling Group Atomically creates and destroys groups of AWS Cloud Resources Amazon SimpleDB App Tier Configures the resources Multi-Tier or Multi-AZ stacks Manages the ordering of provisioning Email ElastiCache Tier Amazon RDS AZ-1 Rolls back in case of failure Or issues AZ-1 Region

26. AWS CloudFormation “Stacks” JSON Template

27. Declarative language

28. { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Create an EC2 instances", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-76f0061f" }, "us-west-1" : { "AMI" : "ami-655a0a20" }, "eu-west-1" : { "AMI" : "ami-7fd4e10b" }, "ap-southeast-1" : { "AMI" : "ami-72621c20" }, "ap-northeast-1" : { "AMI" : "ami-8e08a38f" } } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }, "Outputs" : { "InstanceId" : { "Description" : "InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } }, "AZ" : { "Description" : "Availability Zone of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] } }, "PublicIP" : { "Description" : "Public IP address of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] } } } }

29. Headers { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Create an EC2 instances", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-76f0061f" }, "us-west-1" : { "AMI" : "ami-655a0a20" }, "eu-west-1" : { "AMI" : "ami-7fd4e10b" }, "ap-southeast-1" : { "AMI" : "ami-72621c20" }, "ap-northeast-1" : { "AMI" : "ami-8e08a38f" } } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }, "Outputs" : { "InstanceId" : { "Description" : "InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } }, "AZ" : { "Description" : "Availability Zone of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] } }, "PublicIP" : { "Description" : "Public IP address of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] } } } } Parameters Mappings Resources Outputs

30. Parameters Provision-time specification Command line options

31. "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } },

32. Mappings Conditionals Case statements

33. "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-76f0061f" }, "us-west-1" : { "AMI" : "ami-655a0a20" }, "eu-west-1" : { "AMI" : "ami-7fd4e10b" }, "ap-southeast-1" : { "AMI" : "ami-72621c20" }, "ap-northeast-1" : { "AMI" : "ami-8e08a38f" } } },

34. "Mappings": { "AWSInstanceType2Arch" : { "t1.micro" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, "m1.xlarge" : { "Arch" : "64" }, "m2.xlarge" : { "Arch" : "64" }, "m2.2xlarge" : { "Arch" : "64" }, "m2.4xlarge" : { "Arch" : "64" }, "c1.xlarge" : { "Arch" : "64" }, "cc1.4xlarge" : { "Arch" : "64" } },

35. Dereference this mappings "ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },

36. Resources

37. "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }

38. "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }

39. "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }

40. "KeyName" : { "Ref" : "KeyName" }, Parameter reference

41. "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ] },

42. Map conditional "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ] },

43. "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ] }, Name of map

44. "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ] }, Intrinsic property reference

45. Outputs Returned values

46. "Outputs" : { "InstanceId" : { "Description" : "InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } }, "AZ" : { "Description" : "Availability Zone of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] } }, "PublicIP" : { "Description" : "Public IP address of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] } } } }

47. "AppDatabase": {"Type": "AWS::CloudFormation::Stack", "Metadata": { … }, "Properties": { "TemplateURL": { "Fn::Join": [ "/", [ { … }, "RDS_MySQL_55.template" ] ] }, Embedded Stacks

48. Custom Metadata "Resources": {"Ec2Instance": { "Type": "AWS::EC2::Instance", "Metadata": { "Comment": "This metadata is available via the cfn-describe-stack-resource command line tool or the DescribeStackResource API call", "MyAMI": { "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region“ }, "AMI"]}, "MyRegion": {"Ref": AWS::Region"}, "MyStack": {"Ref": "AWS::StackName"} }, "Properties": { "ImageId": {"Fn::FindInMap": ["RegionMap",{"Ref": "AWS::Region"}, AMI"]}, "UserData": { "Fn::Base64": "80"} } } },

49. Standardized Application Stacks Apache Apache IIS Apache Mongrel Tomcat ASP.NET Mongrel Web Server Rails Struts ASP.NET MVC Rails App Server Your Code Your Code Your Code Your Code MVC logger Log4J Log4Net logger Your Code RubyGems Spring Spring.NET RubyGems Libraries memcached Hibernate nHibernate memcached Packages Ruby Runtime JEE .NET Ruby Runtime DB Caching Centos Linux Windows Centos Framework OS Java Stack .NET Stack RoR stack

50. Bootstrapping Applications with AWS CloudFormation

51. Implement Elasticity 1. Frozen Pizza Model IIS IIS IIS IIS Apache Apache IIS IIS IIS IIS Tomcat Tomcat ASP.NET MVC ASP.NET MVC ASP.NET MVC ASP.NET MVC Struts Struts Your Code Your Code Your Code Your Code Your Code Your Code Log4Net Log4Net Log4Net Log4Net Log4J Log4J Spring.NET Spring.NET Spring.NET Spring.NET Spring Spring nHibernate nHibernate nHibernate nHibernate Hibernate Hibernate .NET .NET .NET .NET JEE JEE Amazon EC2 Windows Windows Windows Windows Linux Linux Java AMI Java Stack

52. Build Job does the following: build the artifact, publish it to Artifactory, build the package, publish the package to the repo . Then there is a follow on job that mounts a base OS image, installs the packages and then creates the final AMI. Source: http://techblog.netflix.com/2011/08/building-with-legos.html

53. Implement Elasticity 2. Take N Bake Pizza Model Apache Your Code Amazon S3 Tomcat Struts Log4J Spring Fetch on boot time Apache Struts Tomcat Source Control Hibernate Your Code JEE Linux Log4J Spring IIS IIS IIS IIS Hibernate IIS IIS IIS IIS JEE .NET .NET .NET .NET Linux Amazon EC2 Windows Windows Windows Windows Golden AMI Java Stack

56. "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bash -ex", "yum -y install git-core", "yum -y install php-pear", "pear install Crypt_HMAC2-1.0.0", "pear install HTTP_Request-1.4.4", "pear channel-discover pear.amazonwebservices.com", "pear install aws/sdk", Bootstrap using User Data

57. Implement Elasticity 3. Made to Order Pizza Model Amazon S3 Apache Apache Struts Tomcat Log4J Hibernate Your Code Spring Tomcat Struts Cookbooks Recipes Source Control Your Code PuppetMaster Log4J Spring Hibernate JEE PuppetClient Agent Linux Linux Windows Amazon EC2 AMI (JeOS) Java Stack

58. Instances ask you a question “Who am I and what is my role?”

60. Simple naming conventions

61. Parallel job execution

62. Consistent serversGreat EC2 Demo : http://docs.puppetlabs.com/mcollective/ec2demo.html

63. Best Practices Puppet is great for incremental implementation! All modules and manifests should be kept under version control. Manage users and groups from the outset. Puppet Environments are your friend Skinny classes, fat modules. Use 'notify' for logging. Make it easy to check logs. ‘The Trifecta‘- Use the Package, file, service.

64. Implement Elasticity 3 approaches to designing your AMIs Easier to Setup Inventory of fully baked AMIs (Frozen/Ready made) “Golden AMIs” with fetch on boot (Take N’ Bake) AMIs with JeOSand PuppetMaster (Made to Order) More Control Easier to maintain

65. More Tools: CloudFormer Create a template from the running resources in your account Select the resources that should be included Customize the logical names Define the template output section Creates a starting point template for your to edit Add parameters Abstract properties and flow properties One-click launch in your account CloudFormer is an appliance that runs in your account

66. One more thing….

67. Optimizing = Cost Savings Free Memory Free CPU Free HDD At 1-min intervals PUT 2 weeks Alarm Amazon CloudWatch Instance Custom Metrics “You could save a bunch of money by switching to a small instance, Click on CloudFormation Script to Save”

69. Elasticity is the fundamental property of the cloud and implement elasticity

70. AWS CloudFormation gives you an easy way to create the set of cloud resources

71. 3 Pizza Models 

73. Still working……

74. Let go of (physical) control but retain your ownership

75. Enterprise Security Features Amazon VPC AWS Identity And Access Management User management Policy-based granular access control Web login to individual users Identity Federation (New!) Multi-Factor Authentication Services Security features Amazon S3 ACL and Bucket policies Amazon EC2 Security Groups, iptables HTTPS API Endpoints

76. SAS 70 Type II Audit ISO 27001/2 Certification PCI DSS 2.0 Level 1-5 HIPAA/SOX Compliance FISMA A&A Low Encrypt data in transit Encrypt data at rest Protect your AWS Credentials Rotate your keys Secure your application Enforce IAM policies Use MFA, VPC, Leverage S3 bucket policies, EC2 Security groups, EFS in EC2 Etc.. In the Cloud, Security is a Shared Responsibility How we secure our infrastructure How can you secure your application and what is your responsibility? What security options and features are available to you?

77. New World Old World Build security in every layer

78. mcollective