In this session, learn how Amazon.com used AWS Database Migration Service (AWS DMS) to migrate 600B+ records to Amazon DynamoDB in two months. We address such questions such as: What were the problems in using Oracle databases for a large-scale distributed system that is constantly growing in scale? How did migration to Amazon DynamoDB address those problems and simplify the application architecture? How do you migrate data reliably and quickly using AWS DMS without affecting your application’s availability or throughput?

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Amazon Migrated Items & Offers for
Retail, Marketplace, & Digital to DynamoDB
Gowri Balasubramanian
Senior Solutions Architect
Amazon Web Services, Inc.
D A T 3 4 7
Uttam Manchikalta
Senior Software Engineer
Amazon.com

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction to Amazon DynamoDB and AWS Database
Migration Service (AWS DMS)
Retail use cases on Amazon DynamoDB
Items and Offers migration to Amazon DynamoDB

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Modern application requirements
Users 1 million+
Data volume TB, PB, EB
Locality Global
Performance Milliseconds, microseconds
Request rate Millions
Access Mobile, IoT, devices
Scale Up and down
Economics Pay as you go
Developer
access
Instant API access
Relational Key-value Document
In-memory Graph

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database characteristics
Relational
Referential integrity
with strong
consistency,
transactions, and
hardened scale
Key-value
Low-latency, key-
based queries with
high throughput and
fast data ingestion
Document
Indexing and storing
of documents with
support for query on
any property
In-memory
Microsecond latency,
key-based queries,
specialized data
structures
Graph
Creating and
navigating relations
between data easily
and quickly
Complex query
support via SQL
Simple query
methods with filters
Simple query with
filters, projections,
and aggregates
Simple query
methods with filters
Easily express queries
in terms of relations
Amazon Aurora
Amazon Relational
Database Service
(Amazon RDS)
Amazon DynamoDB Amazon DynamoDB Amazon ElastiCache
for Redis and
Memcached
Amazon Neptune

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases for internet-scale apps
One of the world’s largest e-
commerce businesses, Amazon, runs
on nonrelational cloud databases
due to scale, performance, and
maintenance benefits
— Werner Vogels
CTO, Amazon

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB
Fully managed nonrelational database for any scale
Fast, consistent performance
Virtually unlimited throughput
Virtually unlimited storage
Encryption at rest and transit
Fine-grained access control
PCI, HIPAA, FIPS140-2 eligible
Service-level agreement
Maintenance-free
Serverless
Auto scaling
Backup and restore
Global tables

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling relational versus non-relational databases
Traditional SQL NoSQL
DB
DB
Scale up
DB
host1
DB
hostn
DB
host2
DB
host3
Scale out to many shards
(DynamoDB: partitions)

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Retail use cases on Amazon DynamoDB
Use cases DynamoDB features
Customer profiles
Shopping cart
Workflow engines
Orders
Reviews
DynamoDB tables with
local/global secondary indexes
DynamoDB streams
AWS Lambda integration to Amazon Redshift,
Amazon Elasticsearch Service, Amazon Simple
Storage Service (Amazon S3)
Promotions, deals/coupons
Adaptive capacity
Auto scaling
DAX
Global tables
Payments (credit cards) Encryption at rest, VPC endpoints

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Commercial data migration and replication software
• Complex to set up and manage
• Application downtime
• Database-engine-specific application code
DB migration challenges
Migration = Cost + Complexity + Time

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are DMS and SCT?
AWS Database Migration Service (AWS DMS) easily and
securely migrates and/or replicates your databases and
data warehouses to AWS
AWS Schema Conversion Tool (AWS SCT) converts your
commercial database and data warehouse schemas to open-
source engines or AWS-native services, such as Amazon
Aurora and Amazon Redshift

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS use cases
Migrate
• Migrate business-critical applications
• Migrate from Classic to VPC
• Migrate data warehouse to Amazon
Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL, or NoSQL to NoSQL
*Targets:
*Refer to AWS documentation for full list of sources and targets
*Sources
Oracle DB
instance
MySQL DB
instance
MS SQL
instance

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Item and Offers Platform

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Item Master Service (IMS)
Blob stores 4
Index tables 7
Databases 24
Partitioning Hash
Logical partitions 256
Record count ~600 Billion
Update rate ~5 Billion/Day
24 Oracle Databases
Sellers
Updates Publish
…..
Website
Oracle DB instance Oracle DB instance Oracle DB instance Oracle DB instance

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges faced

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read-modify-write w/ Concurrency
Continuously update catalog by applying
changes for sellers
Global service with varying peak loads
and I/O requirements
Variable database workloads
Why we choose Amazon DynamoDB?
High throughput bulk access

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
COLUMN NAME TYPE Details
CUSTOMER_ID NUMBER Primary key
SKU VARCHAR2
DOMAIN_ID NUMBER
ITEM_ID VARCHAR2
IS_TOMBSTONED CHAR Active/Inactive
CREATED_BY VARCHAR2 Audit Info
CREATION_DATE DATE
LAST_UPDATED_BY VARCHAR2
LAST_UPDATED_DATE DATE
Oracle schema for index table
SKU to ASIN mapping

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attribute Details
hash_key Partition key
range_key Sort Key
is_tombstoned Active/Inactive
last_updated_by Audit Info
last_updated_date Audit Info
version Version of the
record
DynamoDB schema for index table
SKU to ASIN mapping

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration requirements

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration methodology
Item Master
Service
Oracle
Database
• Application persistence layer is updated with
support for different migration modes that
read or write to Oracle, DynamoDB or both
Oracle
Database
• Using AWS DMS object mapping templates,
Oracle data is transformed to destination
format and written to DynamoDB using
conditional PUTs

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Live migration
Different modes to move data through application
Item
Master
Service
Oracle
Database
Write
Write
Read
Item
Master
Service
Oracle
Database
Write
Write
Read
Read
Dest is based on
lookup store
Item
Master
Service
Oracle
Database
Write
Read
Read

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backfill
Move data using AWS Database Migration Service
Oracle
Databases
AWS DMS
replication
instances
AWS DMS replication
tasks

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS configuration
Selection rule
{
"object-locator":{
"schema-
name":”AMAZON",
"table-
name":"SKU2ASIN"
},
"rule-
action":"include"
}
Object mapping rule
{ "rule-type":"object-mapping", ..., "rule-action":"map-record-to-record",
"object-locator":{ "schema-name":“AMAZON","table-name":"SKU2ASIN"}, "target-
table-name":"SKU2ASIN",
"mapping-parameters":{ "condition-expression":{
"expression":"attribute_not_exists(version) or version <
:record_version",
"exclude-columns":[ "CUSTOMER_ID","SKU", ...],
"attribute-mappings":[ { "target-attribute-name":"hash_key",
"attribute-type":"scalar", "attribute-sub-type":"string",
"value":"{customer_id:${CUSTOMER_ID},sku:"${SKU}"}"}, ... ]}}

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling DMS
Run multiple replication tasks per table
AWS DMS
replication
instance
AWS DMS
replication
instance
AWS DMS
replication
instance
Oracle
Database
Oracle
Database
Oracle
Database
Physical partitions per
table
24
Logical partitions per
table
256
Migration rate per DMS
instance
3K - 5K TPS
Migration rate per table 30K - 100K TPS
Migration rate across all
tables
100K – 150K
TPS
Record count ~ 600 billion
Total data size ~150 TB

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices
• Automate management of replication instances & tasks using AWS DMS SDK
• Design for failure – Test your rollback and recovery mechanisms
• Use optimistic concurrency checks for data consistency
• Fine tune replication task configurations and parallelize to maximize
throughput
• Enable DynamoDB auto scaling and tweak target utilization

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefits
• Availability increased by 10x
• 90% reduction in operations
• Application can scale horizontally
• Simplified persistence layer
• Easier integration with other AWS services

28. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
gowrishb@amazon.com
manchika@amazon.com

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.