Database migration is one of the most commonly performed tasks in the journey to the AWS Cloud. In this workshop, we help migration solutions architects, engineers, and program staff gain in-depth knowledge, hands-on experience, and best practices in using the AWS Database Migration Service (AWS DMS) and its companion, the AWS Schema Conversion Tool (AWS SCT). First, we walk through database migration methodologies and provide field data about what customers are doing with AWS DMS and AWS SCT. Then, we take a closer look at AWS DMS and AWS SCT features, architecture, workflows, and Amazon Aurora as a destination database. Attendees work through a lab to gain firsthand experience in database migration. To get the most out of this session, bring a laptop for the lab and have a QwikLabs account.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GPSWKS408 Announcement:
Qwiklabs Access
To be able to use Qwiklabs, you need a Qwiklabs account. If you don’t have an account,
please create one now at Qwiklabs https://events-aws.qwiklab.com.
Please make sure the email address used for your Qwiklabs account is the same as your
email address used for your re:Invent registration. If not, please send your Qwiklabs email
address to Diego Dalmolin diegodal@amazon.com.

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
GPS: Migrate Your Databases with AWS
Database Migration Service and AWS
Schema Conversion Tool
G P S W K S 4 0 8
J i m H u a n g , A W S P a r t n e r S o l u t i o n s A r c h i t e c t
V i v e k R a j u , A W S P a r t n e r S o l u t i o n s A r c h i t e c t
A s h o k S u n d a r a m , A W S S o l u t i o n s A r c h i t e c t
November 28, 2017

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration is part of
cloud migration journey
How quickly and easily can I migrate my on-premises data to the cloud?
How can I minimize application downtime during database migration?
Are there ways to automate schema conversion and conflict analysis?
How can my databases be merged or modularized during the migration for data
consolidation and application refactoring?

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration patterns
Lift and shift: Move to AWS with as few changes as possible
Leverage Amazon EC2 (e.g., Oracle on Amazon EC2), with a plan for possible changes at a later date
No Amazon RDS or Amazon Aurora
Homogeneous migration: Same database engine with service change
Oracle on premises to Amazon RDS Oracle
Oracle on Amazon EC2 to RDS Oracle
Heterogeneous migration: Migrate to a managed or open-source engine
Oracle to Amazon Aurora, Amazon RDS PostgreSQL, or Amazon RDS MySQL
Migrate traditional RDBMS or commercial data warehouse to Amazon Redshift

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Database Migration Service (AWS
DMS)

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Database Migration Service (AWS
DMS)
AWS DMS along with the AWS Schema
Conversion Tool (AWS SCT) performs
homogenous and heterogeneous database
migration with minimal downtime
Supports most widely used commercial and open-
source databases
Keeps your apps running during the migration
Azure SQL
Amazon
Redshift
Amazon
DynamoDB
Amazon S3
Amazon Aurora

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Schema Conversion Tool (AWS SCT)
Features
Converting database schema
Converting data warehouse schema
Converting application SQL
The AWS Schema Conversion Tool helps automate many
database schema and code conversion tasks when
migrating from source to target database engines

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Endpoints
On-premises Amazon RDS Amazon EC2
On-premises Not supported Supported Supported
Amazon RDS Supported Supported Supported
Amazon EC2 Supported Supported Supported
DMS source/target combinations

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use DMS
Migrate Migrate business-critical applications
Migrate from Classic to VPC
Migrate data warehouse to Amazon
Redshift
Upgrade to a minor version
Consolidate shards into Amazon
Aurora
Migrate from NoSQL to SQL, SQL
to NoSQL, or NoSQL to NoSQL
Amazon S3
Amazon Dynamo
DB
Amazon
Redshift
Amazon S3
Amazon
Aurora
Sources
Targets

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS Architecture, Components,
and Workflows

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Database Migration Service
Replication
Instance & Task
S
O
U
R
C
E
T
A
R
G
E
T
On-premises
database
Amazon RDS
DB instance
M
DB on EC2
instance
On-premises
database
Amazon RDS
DB instance
M
DB on EC2
instance

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS components─replication instances
Performs the work of the migration
Tasks run on replication instances
Can support multiple tasks
AWS DMS currently supports Amazon EC T2 and C4
instance classes for replication instances

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS components─tasks
Run on a replication instance
Contain two and only two endpoints (source and target)
Different migration methods available
Specify selection and/or transformation rules
Support for multiple tasks
Target tables might reside on different databases
You might want to break the migration of a large table
into multiple tasks by using filtering

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS migration methods
Migrating data to the target database
Copy from source to target (migrate existing data)
Capturing changes during migration
Copy from source to target (migrate existing data
and replicate ongoing changes)
Replicating only data changes on the source database
Replicate data changes only

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rules and filters
Selection rules─applied at the source
Include or exclude schemas and tables
Filter rows based on column values
Can use multiple filters
Transformation rules─applied on the target
Rename schemas, tables, and columns
Remove columns
Change case
Add/remove prefix or suffix

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load is table group by table group
Replication
instance
Source Target

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Change data capture (CDC) and apply
Replication instance
Target
Transactions
Change apply after bulk load
t1
t2
Source
t1 t2
Update

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What else can I do?
TargetReplication
instance
Source
Source
Source

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What else can I do?
Target
Replication
instance
Source
Target
Target

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Take it all—or not
Replication
instance
Source Target

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Near-zero Downtime Migration

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer
premises
Application users
AWS
Internet
VPN
Start replication Instance
Connect to source and destination
Select tables, schemas, or databases
Let AWS DMS create tables, load data,
and keep them in sync
Switch applications to new target at your
convenience
AWS
DMS
Keep your apps running during migration

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS: What’s New?
Free DMS
All migrations to Amazon Aurora databases using DMS will be free
Oracle to Aurora PostgreSQL
Cookbook providing detailed instructions on how to convert and migrate your Oracle Databases to Aurora
Database Migration Validation
To ensure that data was migrated accurately from the source to the target, DMS replication task compares
the data on target data store with the data from your source data store
DMS Support for Snowball
A petabyte-scale data transport solution

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Schema Conversion Tool

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Schema Conversion Tool (AWS SCT)
Features
Converting database schema
Converting data warehouse schema
Converting application SQL
Code browser that highlights places where
manual edits are required
Secure connections to your databases with SSL
The AWS Schema Conversion Tool helps automate many
database schema and code conversion tasks when
migrating from source to target database engines

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCT-supported conversions: OLTP
Source database Target database on Amazon RDS
Microsoft SQL Server (version 2008 and later)
Amazon Aurora (MySQL or PostgreSQL), Microsoft SQL
Server, MySQL, PostgreSQL
MySQL (version 5.5 and later) Amazon Aurora (PostgreSQL), MySQL, PostgreSQL
Oracle (version 10.2 and later)
Amazon Aurora (MySQL or PostgreSQL), MySQL, Oracle,
PostgreSQL
PostgreSQL (version 9.1 and later) Amazon Aurora (MySQL), MySQL, PostgreSQL

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCT-supported conversions: Data warehouses
Source database Target database on Amazon Redshift
Greenplum database (version 4.3 and later) Amazon Redshift
Microsoft SQL Server (version 2008 and later) Amazon Redshift
Netezza (version 7.0.3 and later) Amazon Redshift
Oracle (version 10 and later) Amazon Redshift
Teradata (version 13 and later) Amazon Redshift
Vertica (version 7.2.2 and later) Amazon Redshift

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Components of the AWS SCT Console
1. Source schema
2. Action items
3. Target schema
4. Schema element details
5. Edit window

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration assessment
Connect AWS SCT to
source and target
databases
Run assessment report
Read executive
summary
Follow detailed
instructions

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT─other features
SCT Extension Pack
When certain database features can't be converted to equivalent Amazon RDS features, use AWS
services to emulate database functionality
Examples
Oracle send email calls with UTL_SMTP
Microsoft SQL Server jobs with a job scheduler
The AWS SCT extension pack wizard helps you install, create, and configure AWS Lambda functions to
emulate email, job scheduling, and other features

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT─other features (con’t.)
Converting application SQL
When needed with the new database engine, convert the SQL code in your C++, C#, Java, or other
application code. Use AWS SCT to view, analyze, edit, and save the converted SQL code.
Converting data warehouse schema
To optimize the schema conversion:
Choosing strategies and rules
Collecting or uploading statistics
Extracting data for data warehouse migration

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT data extractors
Amazon RedshiftAWS SCT
Amazon S3
Bucket
Extracts through local migration agents
Data is optimized for Amazon Redshift and saved in local files
Files are loaded to an Amazon S3 bucket (through network or AWS Snowball) and then to Amazon Redshift
Extract data from your data warehouse and migrate to Amazon Redshift

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Using AWS DMS and AWS
SCT

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration process
Source
Oracle database on premises
in Amazon EC2 or Amazon RDS
Source
Oracle database on premises
in Amazon EC2 or Amazon RDS
AWS Schema Conversion Tool (AWS SCT)
AWS Database Migration Service (AWS DMS)
Target
Amazon Aurora PostgreSQL
Target
Amazon Aurora PostgreSQL
Step 1
Step 2

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Phase Description Automation
1 Deprecate any objects in source database that are no longer needed
2 Assessment of schema conversion state between source and target SCT
3 Remediation of schema conversion issues (source, scripts, or target) SCT
4 Application conversion/remediation SCT
5 Data migration DMS
6 Data migration validation DMS
7 Performance tuning
8 Deployment
Database migration phases

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon RDS for Amazon Aurora
MySQL 5.6 and PostgreSQL 9.6 (preview) compatible
Scalable with up to 64 TB in single database
Availability: Failover time of < 30 seconds
Highly available, durable, and fault-tolerant custom SSD
storage layer: 6-way replicated across 3 Availability Zones
Single-digit millisecond lag times on up to 15 replicas
Up to 5x better performance than MySQL and up to 2x better
performance than PostgreSQL on the same hardware

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scale-out, distributed, multi-tenant
architecture
Purpose-built, log-structured distributed storage
system designed for databases
Storage volume is striped across hundreds of
storage nodes, distributed over 3 different
Availability Zones
Six copies of data, two copies in each Availability
Zone to protect against AZ+1 failures
Plan to apply same principles to other layers
of the stack
Master Replica
Availability
Zone 1
Shared storage volume
Availability
Zone 2
Availability
Zone 3
Storage nodes with SSDs
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab: AWS SCT and AWS DMS

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab Guide
Objective
Migrate an Oracle database to Aurora
PostgreSQL
Lab setup
Qwiklabs with personal email and
password for login:
https://events-aws.qwiklab.com/
https://events-
aws.qwiklab.com/focuses/73995 (Repeat)
AWS N. Virginia region (us-east-1)
SCT on Windows EC2 instance
Qwiklabs AWS account per student
SCT
Source
endpoint
Aurora
PostgreSQL
AWS
DMS
Target
endpoint
DMS
Task
EC2
EC2
Windows
Remote
Desktop

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab Guide
1. Login Qwiklabs and run DMS Lab
2. Use SCT for Oracle-Aurora schema conversion
Login Windows EC2 instance (EC2-SCT) via Windows Remote Desktop
Connect SCT to Source Oracle and Target Aurora PostgreSQL
Generate and inspect the Database Migration Report
Convert Oracle schema, inspect the conversion results, apply the converted schema to the target database
3. Run DMS for database data migration
Drop foreign keys and disable triggers on the target database
Create source and target endpoints and DMS replication instance
Create and run full-load replication task
Inspect and compare the HR database content in Oracle and Aurora PostgreSQL, respectively
Restore foreign keys on the target database
4. Bonus point – exercise DMS Data Change Capture feature

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS and Migration Best Practices

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT and AWS DMS process
Use AWS SCT to get an assessment report and iterate
through the action items. You need to generate the target
schema multiple times until you come to the final version.
ALWAYS do a database backup before running any
database updates. Make sure to stop all applications that
access the database.
Test your application on your target system.
Practice the actual migration steps and streamline your
process. Each database is unique.

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DMS performance factors
Resource availability on the source
The available network throughput
The resource capacity of the replication server
The ability of the target to ingest changes
The type and distribution of source data
The number of objects to be migrated

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best practices: DMS
Migrate
Load multiple tables in parallel
By default, AWS DMS loads eight tables at a time. You might
see better performance with a larger replication instance
or by adjusting this number
Remove bottlenecks on the target
Try to remove any processes that might compete with each other
for write resources on your target database
Disable unnecessary triggers, validation, and relevant indexes
If migrating to RDS, disable backups and Multi-AZ on the target
until cutover
Use multiple tasks
If you have sets of tables that don’t participate in common transactions,
you might be able to divide your migration into multiple tasks
Enable Logging
Errors, warning and informational messages are written to the logs

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best practices: DMS
Migrate
LOB controls
“Don’t include” – omit LOB columns
“Limited LOB mode” – specify “max LOB size”
“Full LOB mode” – specify “LOB chunk size”
LOB performance
Whenever possible use “limited LOB mode”
If the table has a few large LOB’s and many smaller LOB’s consider breaking it
up into two - a table with the larger LOB’s and another table with the smaller
LOB’s
Migrate them using two separate tasks in “full LOB mode” and “limited LOB
mode”
Optimizing change processing
“Transactional mode” (default) - preserves transactional integrity
“Batch optimized apply” - temporary lapses in transactional integrity possible
“Batch optimized apply” is more efficient as it groups transactions into batches

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Migration Hu b
Simp lify and acce le rate mig ration e x e cu tion
A single location to track the progress of application migrations across AWS and partner solutions
AWS Database
Migration Service

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS database migration partners

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!