Amazon Web Services provides startups with the low cost, easy to use infrastructure needed to scale and grow any size business. Attend this session and learn how to migrate your startup to AWS and make the most out of the platform.
2. What to expect from this session
• Advice for startups migrating to AWS
• A practical checklist
• Relevant AWS services
• A real example from Made.com
3. Services you can use before migrating to AWS
Amazon Cognito
Amazon Mobile Analytics
AWS Device Farm
Amazon SNS
Mobile
Storage & CDN BI & Machine Learning
Email
DNS
Amazon S3
Amazon CloudFront
Amazon QuickSight
Amazon Redshift
Amazon Machine Learning
Amazon Kinesis
Amazon EMR
Amazon Route 53
Amazon SES
Amazon WorkMail
4. Proof of Concept
• Do this early
• Will answer many questions
• Identify gaps and dependencies
• Estimate cost
Start with DEV&TEST!
5. Accelerate your migration with Training
Intro Videos and Labs
• Free introductory training
Online Labs
• Hands-on practice in a sandbox
environment.
Instructor Led Training
• Architecting, Developing, Operating, Big
Data, Security
https://aws.amazon.com/training/
6.
7. Extend your team with AWS Support
• 3 plans to match your needs
• Developer Support
• Business Support
• Enterprise Support
• AWS Infrastructure Event Management
8. Migration approaches
• Big Bang
• Default choice for most startups
• Phased
• Driven by technical complexity
or expiration of old infrastructure investments
• Per application
• Per layer
16. Simplified cutoff process
AWS region
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
17. Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
18. Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
19. Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
DB replication
20. Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
Database
Layer
DB replication
21. Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
Database
Layer
DB replication
23. Cutoff Readiness
• Review Security best practices
• Pen testing
• https://aws.amazon.com/security/penetration-testing/
• Load testing
• Review AWS Service Limits
• AWS Business Support
• A tested Migration process
• A tested Roll-Back process
24. What could possibly go wrong…
• Hard coded IP addresses or host names
• Incorrectly sized AWS resources
• Auto Scaling ramp up period
• High DNS TTL slowing cut over and rollback
• Bandwidth from traditional infra to AWS
• IP address overlap with old network
• Third-party whitelisted IP addresses
• Email limits (request increase/use Amazon SES)
25. Migrating the application servers
• VM Import
• Rebuild
• Configuration management (Chef, Puppet…)
• Docker and the EC2 Container Service (ECS)
• AWS Elastic Beanstalk, AWS Opsworks
• Infrastructure as Code and AWS CloudFormation
27. Empire – PaaS experience on top of ECS
https://youtu.be/8zbbQkszP04
28. Database Migration options
Dump/Restore
• Small database
• Source DB not supporting replication
Native Replication
• Homogeneous migration
• No transformations
AWS Database Migration Service
• Heterogeneous migrations
• Transformations
• Easy to setup
29. Migrating data into AWS
• File transfer using S/FTP, SCP, 3rd party tools
• Point on-premises backup to S3
• AWS Storage Gateway for asynchronous backup
• AWS Import/Export: Disk or Snowball
• Database backup tools
• Database replication tools
• AWS Direct Connect 100 Mbps to 10 Gbps
38. MADE.COM / AWS
How do I migrate my application to the cloud?
Bob Gregory
39. Introduction
Hi everyone, I’m Bob@made.com and I am an Application Architect.
I want to talk about things to do before you migrate, then some things to do
early in migration, and finish by talking about how we are managing cost.
43. Last year we moved our hosting away from:
● Traditional managed service
● 4 big hypervisors (20 core, 320GB RAM)
● 19TB SAN (SAS/SSD auto-tiered)
● All replicated in DR site
● VMWare running about 65 VMs (dev / test / prod)
● Running Magento, OpenERP and Unboxed
44. ...to AWS, because:
● We like being connected to the Internet
● To automate infrastructure build (and get things done faster)
● Autoscaling (vs overprovisioning)
● It’s cool. And this is a better reason than it sounds.
45. Start with something simple
You are going to break things. A big-bang approach will hurt.
46. Start with something simple
If you have no users, you will have no complaints.
Consider starting with Greenfield systems.
47. Start with something simple
How will you integrate your cloud deployments with on-premise?
Loose coupling will save your bacon.
Consider starting with Public HTTP services.
Consider starting with Asynchronous messaging.
48. Start with something simple
It’s easier to handle an outage that only affects you personally.
Consider starting with Development Infrastructure.
50. Before you start
Cloudformation is okay but can be unwieldy.
Consider Terraform if you like JSON.
Consider Ansible if you like extensibility.
51. Before you start
We had several iterations of our network layout.
It is hard to change your entire topology.
Keep it simple.
52. We use 9 subnets.
Address space is cheap.
2 Route tables.
Is this internal or external?
Managed NAT instances
Let AWS worry about it.
53. Before you start
The hardest things to run are stateful services.
Amazon do lots of the hard work for you.
Consider S3.
Consider RDS.
Consider Elasticache.
Consider Elastic File System.
54. Before you start
It is much simpler to manage identical machines.
Treat servers as cattle not pets.
Consider Docker.
55. Before you start
Docker has been a major factor in our success.
Developers can think at an application-stack level.
Teams deliver a working package configuration and a source-
controlled application configuration along with their code-base.
56. Before you start
Keep it simple.
You don’t need to deploy Kubernetes on top of Mesos
+ =
57. Before you start
There is no charge for running extra accounts.
Set up Consolidated Billing and run multiple AWS accounts.
58. We use
A hub-and-spoke model.
We run a vpn in Management.
VPC Peering
to access other environments
Separate Production.
This allows easier access
control
A separate account for
testing automation
59. We use
A Bastion account for IAM.
Like sudo for AWS.
Two-factor authentication
to access other environments.
Role-based access.
Developers are only allowed
to destroy test machines.
http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
62. A 30 second Ansible tutorial:
Please can I have exactly one
instance with the Name
“my-instance”?
63. A 30 second Ansible tutorial:
Please can I have exactly one
instance ?
64.
65. Don’t be this guy
It’s called best practice for a reason
66. So you’ve deployed your first system.
You’ve chosen your automation tooling
You’ve pushed data into stateful backing services
You’ve containerised your codebase and its configuration
68. Before you deploy your second service
Instances come and go. In order to manage your environment you
will need telemetry.
Monitor all the things.
Create Health Dashboards for each system you deploy.
69. Before you deploy your second service
Cloudwatch is okay.
It’s easy to set up and
free for basic
monitoring.
70. We use
Collectd
To gather metrics
Riemann
To process and alert
InfluxDB
To store metrics
Grafana
To make pretty
pictures
71. Before you deploy your second service
Instances come and go. In order to manage your environment you
will need log shipping.
Log all the things.
Developers shouldn’t need to ssh to instances just to view logs.
72. We use
Rsyslog
To gather and process
logs
Logstash
To route logs into indexes
Elasticsearch
To store logs
Kibana
To view logs
73. This is expensive in engineering time
You must implement monitoring, but there are quicker ways to get
going.
Consider SAAS.
74. Time to move the big things
Database migration is the worst part (except for all the other worst
parts).
Amazon’s Database Migration Service is essentially magic.
Find it in the RDS tab of the AWS Console.
75. We used
HAProxy on the old and
new systems.
To switch over, we
configured HAProxy OLD to
route traffic to HAProxy
NEW.
To fail back, we could
configure HAProxy NEW to
route to HAProxy OLD.
76. So where are we now?
We prioritised agility over cash.
Each service has redundancy, and
shares no infrastructure with
other systems.
This makes it simple for developers
to deploy their stacks.
77. So where are we now?
We prioritised agility over cash.
Each service has redundancy, and
shares no infrastructure with
other systems.
This makes it really expensive.
78. How are we reducing costs?
Not everything needs to be up all the time
Consider Turning off test environments overnight.
Consider Scaling down database instances.
Consider Auto-scaling Groups.
79. How are we reducing costs?
On-demand instances are the expensive option.
Consider Reserved Instances for RDS, Elasticache, anything
where you have some well-known capacity.
80. How are we reducing costs?
On-demand instances are the expensive option.
Consider Spot Instances for automated tests and batch
processing.
81. How are we reducing costs?
CloudHealth has useful reports on instance over-sizing;
unattached volumes; and violations of best practice.
They charge a fixed percentage of your bill, so you are never too
small to consider using them.
82. How are we reducing costs?
The next step is Container Scheduling.
By abstracting away the ec2 instances, we can retain agility while
deploying multiple systems to each instance.
By using servers more efficiently we can cut our bill.
83. How are we reducing costs?
Container scheduling requires
Great monitoring of service performance and health.
Log shipping from your containers.
Service discovery (we’re using consul)
A fundamental rethink of the way you architect systems.
It has taken us approximately a year to reach this point.