The power of AWS cloud needs to be understood to be harnessed in the most effective manner. This first Winnipeg AWS User Group meetup provides a forum to explore the technology approach delivering successful solutions on AWS.
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Architecting for AWS Cloud - let's do it right!
1. ARCHITECTING FOR AWS CLOUD - LET'S DO IT RIGHT!
Misha Hanin
200 – 5 Donald Street, Winnipeg, MB, R3L 2T4, Canada | info@iRangers.com | P: 1.855.996.4742
Solutions Managing Director
misha.hanin@irangers.com | @mishahanin
2. How Familiar Are You With AWS?
Never Heard Of It
Considering
Using
Expert
3. Why Do I Care?
• AWS Certified Solutions Architect
• One of the nearly 500 trained Microsoft Certified Masters (MCM) in the
world (during 10 years existence of MCM program, just about 500 people in the world participated
in this very intensive training, ONLY 5 from CANADA)
• Working with Cloud technologies since the early 2004, beginning with
Google, RackSpace, The Planet (SoftLayer), etc.
• Working with Microsoft stack since the early 90’s, beginning with
Windows 1.0
• Many Exchange and AD deployment projects, including Office 365
migrations
• Working with Office 365 since early betas, when it was known as
Live@Edu (2010)
4. Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
What is AWS?
6. 13 Regions 35 AZ’s
An independent collection of AWS resources in a
defined geography
A solid foundation for meeting location-dependent
privacy and compliance requirements
Global infrastructure
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
Global infrastructure
The AWS Cloud infrastructure is built around
Regions and Availability Zones (“AZs”). A Region
is a physical location in the world where we have
multiple Availability Zones.
7. 54 Edge Locations
Supports global DNS infrastructure (Route53) and
Cloudfront CDN
Edge Locations
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
9. Networking
AWS networking shares a lot in common with the
way we run things locally + some extras!
• Amazon Virtual Private Cloud
(Amazon VPC)
• VPN & Direct Connect
• Elastic Load Balancing
• Amazon Route 53
Networking
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
10. Route 53
Highly available and scalable Domain Name System
Extremely reliable and cost effective
Networking
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Health Checks and DNS Failover
• ELB Integration and Failover
• Zone File Imports
• CloudWatch
• Health Check Graphs
11. Route 53
Highly available and scalable Domain Name System
Extremely reliable and cost effective
Networking
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
13. Vertical Scaling
From $0.02/hr
Elastic Compute Cloud (EC2)
Basic unit of compute capacity
Range of CPU, memory & local disk options
40 Instance types available, from micro to cluster compute
Compute
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Price Reductions
• Free Tier & 64bit AMIs for RHEL & SUSE
• Amazon Linux AMI
• VPC for Everyone
• More Flexible IP addresses in VPC
• More EBS-optimized Instance Types
17. Elastic Block Store
High performance block storage device
1GB to 1TB in size
Mount as drives to instances
Storage & CDN
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Incremental Snapshot Copy
• AMI Copy between Regions
• 4000 Provisioned IOPS per Volume
18. S3 - Durable storage, any object
99.999999999% durability of objects
Unlimited storage of objects of any type
Up to 5TB size per object
Storage & CDN
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Lower Request Pricing
19. Storage Gateway
Connecting on-premises IT environments with cloud storage
Gateway-cached volumes
Gateway-stored volumes
Storage & CDN
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Hyper-V Support
20. CloudFront
World-wide content distribution network
Easily distribute content to end users with low latency,
high data transfer speeds, and no commitments.
London
Paris
NY
Served from S3
/images/*
3
Served from EC2
*.php
2
Single CNAME
www.mysite.com
1
Storage & CDN
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Custom SSL Domain Names
• Root Domain Hosting
23. Relational Database Service
Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
Database
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• General Availability and SLA
• 3TB, 30 000 IOPS
• MySQL 5.5 Major Version Upgrade
• MySQL 5.6 Support
• Oracle Data and Network Encryption
• SQL Server Major Version Upgrade
• Easy Access to Log Files
• Read Replica State Monitoring
25. ElastiCache
Memcached compatible caching layer
Serve frequently requested & slow changing
data from scalable clusters
Reduce load on database and other servers
Database
• Enhanced Cache Nodes (M3) in All
Regions (except GovGloud)
• Reduced Prices in US West and
South America
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
28. Application Services
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
Simple Email Service
Amazon Simple Email Service (Amazon SES) is a cost-
effective email service built on the reliable and
scalable infrastructure that Amazon.com developed to
serve its own customer base.
• High Deliverability
• Multiple Email-Sending Interfaces
• Sending & Receiving Statistics
• Notifications
29. Application Services
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
Simple Notification Service
Fast, flexible, fully managed push messaging service
Push a message once, deliver it one or more times
Group multiple recipients using topics
• Push Notifications to Mobile Devices
• Amazon
• Apple
• Google
• 256KB Payloads
30. Task A
Task B
(Auto-scaling)
Task C
2
3
1
Simple Workflow Service
Reliably coordinate processing steps across
applications
Integrate AWS and non-AWS resources
Manage distributed state in complex
systems
Application Services
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Flow Framework for Ruby
• gem install aws-flow
31. Application Services
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
Elastic Transcoder
Easy-to-use scalable media transcoding
Cost-effective, Managed, Secure
Seamless Delivery with integrated AWS Services
• Apple HTTP Live Streaming (HSL)
• WebM
• MPEG2-TS
• Multiple Outputs per Job
• Automatic Bitrate Optimization
• Watermarking
• Enhanced S3 Output Options
33. Elastic Beanstalk
One-click deployment from Eclipse, Visual Studio and Git
Rapid deployment of applications
All AWS resources automatically created
Deployment & Management
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• New Management Console
• VPC, RDS and Configuration Files
• IAM Roles
34. CloudFormation
Automate creation of ‘stacks’ in a repeatable way
Scripting framework for AWS resource creation
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
Deployment & Management
• Parallel Stack Processing
• Nested Stacks
• Support for several additional
AWS resource types
36. Identity & Access Management
Granular control of user rights with AWS
Automated granting of service rights
Deployment & Management
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Management
Networking
• Resource-level Permissions for EC2
Security Groups
• Amazon, Facebook, Google Identity
Federation
• Variables in Access Control Policies
40. Thank you
Misha Hanin
200 – 5 Donald Street, Winnipeg, MB, R3L 2T4, Canada | info@iRangers.com | P: 1.855.996.4742
Solutions Managing Director
misha.hanin@irangers.com | @mishahanin
Hinweis der Redaktion
Before we start, let see who is in the room. How Familiar Are You With AWS?
This is a simple view of the set of services that AWS offers. At the core are the building block utility services – for compute, storage and data. AWS then surrounds these offerings with a range of supporting components like management tools, networking services and application services. All this is hosted within AWS global data center footprint that allows you to consume services without having to build or manage the infrastructure yourself
Let’s take a look at some of the innovations AWS has delivered, over the past years
AWS Global Infrastructure
The AWS Cloud operates 35 Availability Zones within 13 geographic Regions around the world, with 9 more Availability Zones and 4 more Regions coming online throughout the next year.
https://aws.amazon.com/about-aws/global-infrastructure/
As of today, AWS has a total of 54 Edge locations. They support AWS’s global DNS service Route53, as well as Amazon’s Cloudfront CDN.
This has helped AWS’ customers in or near those countries, enjoy lower latencies and better user experience with content downloads.
Now let’s take a look at compute
Amazon Virtual Private Cloud (Amazon VPC) enables us to launch Amazon Web Services (AWS) resources into a virtual network that we've defined. This virtual network closely resembles a traditional network that we'd operate in our own data center, with the benefits of using the scalable infrastructure of AWS.
Now let’s move to the networking layer. Amazon Route 53 is Amazon’s fast, fully managed DNS service. It lets us easily host our domain names and zones. Route 53 DNS servers will reply to our domain name queries with low latency, and direct them to our service endpoints – for example to our Elastic Load Balancers, Cloudfront distributions or S3 buckets.
The first new feature is DNS failover. Now we can configure Route53 to perform health checks of our website’s availability, and in case our site has issues, Route 53 can direct our traffic to an alternate site. For example another copy of our system, or to a static S3 website while we fix our primary, dynamic system.
For ELB endpoints, Route 53 evaluates the health of the load balancer itself and the health of our application running on the EC2 instances behind it. If any part of the stack goes down, Route 53 detects the failure, routes traffic away from the load balancer, and directs traffic to other healthy ELB endpoints. Route 53 DNS Failover also supports EC2 endpoints as well as endpoints located in our own datacenter.
Route 53 also reports these metrics in Cloudwatch. Here’s an example of a health check graph of two monitored websites
Here’s an example cloudwatch healh check graph of two website endpoints monitored by Route53. Note that the failures were self-induced ;)
Now let’s take a look at compute
Which is implemented by our Elastic Compute cloud, or EC2. There have been lots of announcements for EC2 recently.
AWS has had several price reductions, including –
An up to 27% reduction for Linux reserved instance prices for standard m1, second gen standard m3, high mem m2, and high cpu c1 instance families.
Now Reserved instances provide savings of up to 65% compared to on-demand instances. You should definitely explore using RI’s especially for your heavy utilization base load. One handy way to check if they would help you, is to leverage the AWS Trusted Advisor free trial, and let it audit your system to optimize for cost, availability and security
An 80% price reduction for Dedicated Per Region Fee –
A reduction of up to 37% for Dedicated On-Demand Instances and
A reduction of up to 57% for Dedicated Reserved Instances –Dedicated Reserved Instances also provide additional savings of up to 65% compared to Dedicated On-Demand instances.
Let’s talk about VPC. VPC lets you create a virtual network of logically isolated EC2 instances and an optional VPN connection to your own datacenter. We want every EC2 user to be able to benefit from the advanced networking and other features of Amazon VPC. To enable this, instances for new AWS customers (and existing customers launching in new Regions) will be launched into the "EC2-VPC" platform.
You don’t need to create a VPC beforehand - simply launch EC2 instances or provision Elastic Load Balancers, RDS databases etc like you would in EC2-Classic and we’ll create a VPC for you at no extra charge. We’ll launch your resources into that VPC and by default assign each EC2 instance a public IP address. The option of allocating a default public IP address is also now configurable. With default VPC, You can start taking advantage its features, such as assigning multiple IP addresses to an instance, changing security group membership on the fly, and adding egress filters to your security groups. However the default VPC behavior is compatible with EC2 classic, so everything should work as before for your systems.
These 2 screenshots show how you can see an account where the new EC-VPC is configured and in use.
However, If you’ve previously launched an EC2 instance in a Region or provisioned ELB, RDS, or ElastiCache in a Region, AWS won’t create a default VPC for you in that Region. If that is the case for you and you want to start using default VPC, you have two options. You can create a new AWS account or you can pick a Region that you haven't used (as defined above)
Elastic Load Balancing (ELB) supports Proxy Protocol version 1. We can now identify the originating IP address of a client connecting to our servers using TCP load balancing. It simply prepends a human readable header with the client’s connection information to the TCP data sent to our server.
The EBS Snapshot Copy feature gives us the power to copy EBS snapshots across AWS Regions. Now AWS has made the snapshot copy much faster with support for incremental copies
The first time we copy an EBS snapshot of a volume to another Region, all of the data will be copied. Subsequent copies will be incremental: only the data that has changed since the last copy will be transferred. Based on our findings, we expect to see up to 50x speedup for the incremental copies of an EBS volume snapshot.
The AMI Copy feature leverages this, and makes replicating your AMI’s between regions significantly faster. This is a handy way to have a fast, reliable and repeatable way to replicate your application building blocks across multiple regions.
AWS has also increased the provisioned IOPS maximum to 4000 I/Os per second, and up to 1TB per volume. Provisioned IOPS volumes are designed to deliver predictable, high performance for I/O intensive workloads such as databases, and enterprise applications. We should definitely use EBS optimized instance types in together with provisioned IOPS.
Good news for all S3 users. AWS has reduced S3 request prices in all regions. AWS are lowering the prices for GET requests by 60% and the prices for PUT, LIST, COPY, and POST requests by 50%.
The AWS Storage Gateway allows us to marry our existing on-premises storage systems with the AWS cloud for backup, departmental file share storage, or disaster recovery. Now AWS has added support for running the gateway appliance on Microsoft’s Hyper-V environment.
We can now use the Storage Gateway on-premises in with either Hyper-V or VMware ESXi, Or we can run the Storage Gateway appliance on Amazon EC2.
AWS’ CDN service Amazon CloudFront now supports two new frequently requested features: support for custom SSL certificates and the ability to point the root of our website to a CloudFront distribution. With support for both of these features, it is now even easier for us to deliver our entire website via CloudFront’s global network of edge locations.
To use custom SSL certificates, we need to
Purchase a Certificate from a Recognized Certificate Authority.
Upload the Certificate to our AWS Account.
Map our Domain Name to Your Distribution.
Note that there is a fixed monthly fee for each custom SSL certificate, with pricing pro-rated to each hour of usage. More information on the pricing, please see the CloudFront pricing page.
On this screenshot we can also see cloudfront hosting a root domain, in other words the domain name cloudfrontdemo.com, without any prefix like www. It’s achieved by configuring an Alias, or A record that maps to the apex or root of our domain. Once configured, Route 53 will respond to each domain name request with the IP address of this cloudfront distributions. That way our users don’t need to specify www. Prefixes, and we don’t have to use redirects which can slow down the content access
With strong customer adoption across multiple market segments, numerous new features, and plenty of operational experience behind us, we also have a Service Level Agreement or SLA), for Amazon RDS, with 99.95% availability for Multi-AZ database instances on a monthly basis. This SLA is available for Amazon RDS for MySQL and Oracle database engines because both of those engines support Multi-AZ deployment.
30k / 3TB on MySQL and Oracle
Mysql 5.x major version upgrade – modify DB, change version to 5.x. Best practice is to create a snapshot, make a new RDS from the snapshot, modify and upgrade the snapshot and test it. Before repeating this for your production DB.
Binary Log Access - You can download and stream binary logs through the native mysqlbinlog tool. This can be useful for a variety of purposes such as syncing data with an on-premises deployment, audit logging, analytics, and debugging of replication errors.
For troubleshooting, it’s now very handy to view the logs as of a certain point in time, watch them for real-time updates, or download them. The downloads can be made via the rds-download-db-logfile command
AWS has fine-tuned the storage and there processing model, optimized replication pipeline, and taken advantage of the scale to drive down our hardware costs.
As a result, AWS has reduced the prices for Provisioned Throughput Capacity (reads and writes) by 35% and Indexed Storage by 75% in all AWS Regions
Furthermore, If you are able to predict your need for DynamoDB read and write throughput in an AWS Region, we can save even more with a new Reserved Capacity pricing model. If you need at least 5,000 read or write capacity units over a one or three year time period we can now enjoy savings that range from 54% to 77% when computed using the newly reduced On-Demand pricing. The net reduction with respect to the original pricing works out to be 85% lower costs
Building a large-scale email solution is often a complex and costly challenge for a business. You must deal with infrastructure challenges such as email server management, network configuration, and IP address reputation. Additionally, many third-party email solutions require contract and price negotiations, as well as significant up-front costs. Amazon SES eliminates these challenges and enables you to benefit from the years of experience and sophisticated email infrastructure Amazon.com has built to serve its own large-scale customer base. Amazon SES has a range of features that make it the ideal solution for sending and receiving email.
Push notifications are short, alert-style messages we can send to users even when they are not actively using our app. The experience is similar to SMS, but it costs much less because it uses Wi-Fi or cellular data. Users can choose to acknowledge a push notification to launch our app and see more information.
Implementing push notifications can be tricky, especially when we target multiple platforms such as iOS, Android and Kindle Fire.
Customers tell us that this is just the sort of undifferentiated heavy lifting they like us to solve on their behalf. AWS is enhancing the Amazon Simple Notification Service with Mobile Push, a new feature that transmits push notifications from backend server applications to mobile apps on Apple, Google and Kindle Fire devices using a simple, unified API. We can send a message to a particular device (direct addressing), or we can send a message to every device that is subscribed to a particular SNS topic (broadcast).
The Amazon Simple Workflow Service (SWF) lets you build scalable, event-driven systems that coordinate work across many machines that can be either cloud-based or on-premises. The service handles coordination, logging, and auditing so don't need to write glue code or to maintain our own state machines. We can focus on the business logic that adds value to our business.
Apple HTTP Live Streaming (HLS) Support. Amazon Elastic Transcoder can create HLS-compliant pre-segmented files and playlists for delivery to compatible players on iOS and Android devices, set-top boxes and web browsers.
WebM Output Support. Amazon Elastic Transcoder can now transcode content into VP8 video and Vorbis audio, for playback in browsers, like Firefox, that do not natively support H.264 and AAC.
MPEG2-TS Output Container Support. which are commonly used in broadcast systems.
Nested stacks, for each of the application tiers. Here the top or service-level stack depends on 3 different application tier stacks.
Furthermore, Modifying a top-level stack cascades the modification only to those nested stacks which require changes
Variables, so you can have processing logic in our IAM policies, for example to target rules per user name, instead of having a policy for each user name separately
HSM is short for Hardware Security Module. It is a piece of hardware -- a dedicated appliance that provides secure key storage and a set of cryptographic operations within a tamper-resistant enclosure. We can store our keys within an HSM and use them to encrypt and decrypt data while keeping them safe and sound and under our full control. We are the only one with access to the keys stored in an HSM.
For more information please visit the CloudHSM pages at aws.amazon.com