Discuss data migration with AWS experts - STG304 - Santa Clara AWS Summit

© 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Chalk Talk: Discuss data migration
with AWS experts
Paul Reed
Principal Product Manager
AWS
S T G 3 0 4
Sai Maddali
Senior Product Manager
AWS
Dylan Locsin
Senior Product Marketing Manager
AWS

Agenda
Cloud data migration: Key questions & considerations
AWS portfolios for storage and data transfer
AWS DataSync – What it is and how it works
AWS Snow family – What it is and how it works
Network transfer rates
Q&A on use cases – preferably yours

Data migrations – Key questions
Why are you and your teams moving data to AWS?
What is the data you’re moving?
Where is the data going, to which service, for what application & use case?
When do you need the data off your array/out of the DC/in AWS?
How much data? How much usable network capacity?
How you move the data depends on all of the above.

Storage portfolio breadth and depth
Amazon Elastic
Block Store
(Amazon EBS)
Amazon S3 Amazon S3 Glacier
Data movement
OnlineOffline
AWS Snow family
AWS Storage Gateway family
AWS Direct Connect
AWS DataSync
Amazon S3 Transfer Acceleration
Storage partners
AWS Transfer for SFTP
Amazon Kinesis Data Streams, Video
Streams, & Firehose
Amazon EFS
Databases
Amazon RDS
Amazon Aurora
Amazon Redshift
Amazon DynamoDB
Amazon ElastiCache
Amazon QLDB
Amazon Neptune
Amazon Timestream
Amazon DocumentDB
AWS Database Migration Service
Storage & database portfolio breadth and depth
CloudEndure

How? AWS data transfer & migration portfolio
Online
data transfer
Hybrid
storage
Offline
data transfer
Private
network
connections
to AWS
Load streaming
data into
Amazon S3
Ship static data
into and out of
Amazon S3
Access AWS
storage from
on-premises
Edge locations
for Amazon S3
enabled
applications
Accelerated
transfer of
active data
AWS
DataSync
AWS
Transfer
for SFTP
SFTP transfers
into Amazon
S3
Storage and
compute in
disconnected
environments
Migrate live
apps. &
databases
NEW NEWNEW NEW

Backup & restore Archive Primary storage BC/DRData migration
AWS Partner Network (APN): Migration & storage

What use cases need to transfer active data?
Migration of active
applicationdata
Transferring data for
time sensitive in-
cloud analysis
Replication of data
for business
continuity
Online data transfer

What is AWS DataSync?
Online transfer service that simplifies, automates, and accelerates
moving data between on-premises storage and AWS services
Fast data
transfer
Cost-
effective
Enterprise
ready
Combines the speed and reliability of network acceleration software
with the cost-effectiveness of open source tools
Easy to use Secure and
reliable

AWS DataSync
Up to 10 Gbps with inline compression, sparse file detection, and incremental transfers.
Optimized read/write for Amazon S3 and AmazonEFS.Fast data
transfer
Easy to use
Secure and
reliable
Enterprise
ready
Cost-
effective
Manage through the AWS Management Console, AWS CLI, or SDK, monitor with AmazonCloudWatch.
Service infrastructure auto-deployed and on-premises agents automatically updated.
Datavalidated and encrypted in-transit and at-rest.
Automatic recovery from I/O or transmission errors.
Bandwidth management, IAM, and AWS CloudTrail integrated.
PCI-DSS compliance, HIPPAA eligible.
$0.04 per-GB copied

As simple as one, two, three ...
NFS file systems,
Amazon S3 buckets, or
Amazon EFS file systems
Create a task by configuring
storage locations and options
Monitor in the
console or through
Amazon CloudWatch
Start the task and monitor
progress
VMware ESXi or Amazon EC2
8 vCPU
32 GB memory
Deploy the agent and
associate to your AWS account
1 2 3

Shared
file system
NFS TLS
How AWS DataSync works
On-Premise
Amazon S3
bucket
AWS Storage resources
AWS
DataSync
Agent deployed for
fast access to local
storage
Region
Amazon EFS
file system
AWS DataSync
agent
Data transfer via
efficient purpose-
built protocol
Managed from the
console or AWS CLI
Service in AWS
writes or reads
data from AWS
storage services

Reduce infrastructure with AWS DataSync
and AWS Storage Gateway
AWS DataSync transfers to Amazon S3 as part of migration incremental sync keeps Amazon S3 up-to-date1
2 Latency-sensitive applications can access data through file gateway, can decommission NAS
NAS
Amazon S3
bucket
On-Premise Region
Applications 2
AWS DataSync
AWS DataSync
agent
1

AWS Snow family: Move large volumes into the cloud
Moving large volumes of data over the internet can take
a long time
The cloud is not always accessible from remote locations
where connectivity is limited or intermittent
Traditional shipping of conventional hard drives is
laborious and error prone
AWS Snowball AWS Snowball Edge

AWS Snowball AWS Snowmobile
• 80-TB storage capacity
• 10GE networking
• Dataencryption end-to-end
• Rugged 8.5-G impact case
• Rain and dust-resistant
• Small file batching
• Snowball command line
• S3 adapter runs on workstation
• Exabyte-scale storage in a 45-ft
container
• Dedicated security personnel
• GPS tracking, alarm monitoring, 24/7
surveillance, and optional additional
security
AWS Snow family
• 42 or 100-TB storage capacity
• Rugged 8.5-G impact case
• Rain and dust resistant
• Amazon EC2/AMI support for edge
compute AND migration
• Clustering
• AWS IoT Greengrass support
• Optional GPU
AWS Snowball Edge
Compute or Storage Optimized

Create
a job
Device gets shipped
Copy data and ship the
device back
Your data is loaded into
your bucket!
In transit Delivered to you At AWS Sorting Ingest to AWS
Job created
Job completed
1–3 Days
Data transfer in 3 steps with Snowball devices
Job Provisioned
1–2 Days 1-N Days (file transfer speed dependent) 1–2 Days 1–N Days
At AWS Region

Job creation
Customer premises Region
AWS console Snowball
service

Data transfer
Customer premises Region
AWS console Snowball
service
Local
infrastructure
Snowball
service
S3 bucket
S3 bucket

Copy options for Snowball and Snowball Edge
Workstation Snowball
Snowball client
Workstation
Snowball
(S3 adapter)
Workstation
S3 API
Workstation
Workstation
S3 API / NFS
Workstation
Snowball
Edge
Snowball client

Snowball Edge: For edge computing (e.g., analytics, machine
learning) in remote, disconnected and harsh environments
Various industries ranging from research to transportation necessitate edge computing
capabilities to help cope with environments where data generation is decentralized,
data volumes are significant, and network connectivity is either inaccessible or
intermittent.
Research Industrial Healthcare Transportation

Amazon Simple Storage
Service (Amazon S3)
Compatible Endpoint
NFS File Interface
Storage cluster AWS IoT Greengrass and
Lambda support
Ruggedized enclosuresHardware enabled data
encryption
Capabilities that extend Snowball Edge to the “edge”
Amazon Elastic Compute
Cloud (Amazon EC2)
Instance support

Snowball Edge - Options
Amazon S3 Compatible Storage 42 TB 100 TB
Compute Comparable to m5d.2xlarge Comparable to m4.4xlarge
AWS Services available Amazon Lambda, File Gateway, Amazon EC2 Lambda, File Gateway, Amazon EC2
Memory 208 GB 32 GB
vCPUs 52 24
Disk Space for Instances 7.62 TB 1 TB
Clustering Available Available
Typical job lifetime Can be long lasting About 1 month
Networking Max 100 Gb Max 40 Gb
GPU Optional Nvidia Tesla V100 No
Storage
Optimized
Compute
Optimized

Network transfer rates & transfer methods

Use case example 1 – Remote site retirement

Use case 2 – NAS migration
AWS

Use case 3 – Video collection at the edge

Use case 4 – Hybrid data processing flows
AWS

Use case 5 - Migrating media archives

Related recorded webinars – AWS Online Tech Talks
Introduction to New AWS Services for Data Transfer
Data Transfer Services: AWS Snow Family Briefing
Introduction to Amazon FSx for WindowsFile Server
aws.amazon.com/about-aws/events/monthlywebinarseries/on-demand/*
*Also available onYouTube
Best Practices forMigrating Oracle Databases tothe Cloud

Thank you!
S UM M I T © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.

Amazon Simple Storage
Service (Amazon S3)
Compatible Endpoint
NFS File Interface
Storage and compute
cluster
AWS Greengrass and
Lambda support
Line rate data transfer
Hardware enabled data
encryption
Snowball Edge key features
Amazon Elastic Compute
Cloud (Amazon EC2)
Instance support

AWS Direct Connect
Reduce bandwidth
Costs
Consistent
network
performance
Compatible with
all AWS services
Private
connectivity to
VPC
Elastic Simple
• Establishprivate connectivitybetween AWS and your data center
• Dedicated connectioncan be partitioned into multiple virtual interfaces
• Maintain network separation between public and private environments
BENEFITS

Simplify and Accelerate Migration With CloudEndure
• Designed for rapid, mass-scale migrations
• Simple setup lets you start in minutes
• Same highly automated process for any workload
(regardlessof OS type/version,application,or DBs)
• Eliminates complexityand reduces risk
• Migrate with minimal business disruption
Available in the AWS Marketplace
& included for MAP-qualified customers
and Migration Competency Partners
Highly Automated
MigrateFrom
Any Source
Short Cutover Windows
With Minimal Downtime
Minimal Skill Set
Required to Operate
Easily Plugs Into
Migration Factories
and Cloud COEs
Easy, Non-Disruptive
Tests Prior to Cutover
ReliableFlexible
Option to
MigrateBack
Robust, Predictable, Non-
DisruptiveContinuous
Replication
Wide Range of OS,
Application, and
DatabaseSupport
Highly Secure
for Regulated
Environments

How CloudEndure Migration Works
Lightweight Staging Area in
Target Region
Source
Data Center
Target Subnet(s)
Continuousdata
replicationtraffic
(compressedand
encrypted)
Lightweight Linux
Replication Server
Low-Cost
Block Storage
Orchestration& system
conversionreadyto run
workloadsin minutes
regardlessof disk size
• CloudEndure continuously replicates any application or database from any source into AWS
• Business outcome: Allow self-service, rapid, reliable migrations with minimal business disruption

Physical
Data Centers
Wide Platform Support*
Any
Application
Any
Database
x86 Operating
Systems
Source
Infrastructure
* See documentation or contact Support for a complete list.

CloudEndure License model and procurement path
•Licensed per unique source machine migrated
•Purchase on AWS Marketplace with self-service setup
•For assistance, contact your consulting partner or AWS professional services
•Benefits available to MAP qualified customers and Migration Competency Partners

How CloudEndure Disaster Recovery Works
Lightweight Staging Area in
Target Region
Source
Data Center
Target Subnet(s)
Continuousdata
replicationtraffic
(compressedand
encrypted)
Lightweight Linux
Replication Server
Low-Cost
Block Storage
Orchestration& system
conversionreadyto run
workloadsin minutes
regardlessof disk size
• CloudEndure continuously replicates any application or database from any source into AWS
• Business outcome: Use AWS to significantly reduce DR TCO, while improving recovery objectives

Amazon S3 Transfer Acceleration
AWS Edge
Location
Optimized
Throughput!
Change your endpoint, not your code
Leverages Amazon CloudFront
global edge locations
Optimized protocols
No firewall exceptions
No client software required

Getting Started
Enable Amazon S3 Transfer Acceleration for their bucket
DNS resolution
<bucket>.s3-accelerate.amazonaws.com resolves to the nearest
POP location for the client
uses the client IP and CloudFront latency based routing
Update application / destination URI to:
<bucket-name>.s3-accelerate.amazonaws.com
File is uploaded to:
<bucket-name>

Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore
Time[hrs]
500 GB uploadfrom these edge locationsto a bucket in Singapore
Public Internet
How Fast is Amazon S3 Transfer Acceleration?
S3 Transfer Acceleration
On average, we have seen 171%
improvement over regular Amazon S3
when uploading over long distances

AWS Storage Gateway
Existing Data
Archives
New Data
Corporate
Data Centers or
Offices
AWS Storage Gateway
HTTPS
NFS
SMB
iSCSI
VTL
Hybrid storage service enabling applicationsto seamlessly use AWS storage
On-Premises Gateway Service
Amazon S3
Amazon EBS
snapshots
Amazon Glacier

The AWS Storage Gateway family
File gateway
Store and access objects in Amazon S3 from file-based
applications with local caching
Volume gateway
Block storage on-premises backed by cloud storage with local
caching, Amazon EBS snapshots, and clones
Tape gateway
Drop-in replacementfor physical tape infrastructure backed by cloud
storage with local caching
Three gateway types provide file, block, and tape storage interfaces

AWS Storage Gateway key features
Standard storage protocols
Seamlessly connect on-premisesapplications
to AWS storage without modifyingapplications
Fully managed cache
Frequently used datacached locally
For low-latency access
Optimized data transfer
Securesupload of changed data and
downloadsrequested data
AWS integrated
Manage on-premise gatewaysin AWS Management
Console and use your data across the entire portfolio of
AWS applicationsand services

Some AWS Storage Gateway customers

File gateway overview
Store and access objects in Amazon S3 from file-based applications with local caching
Use cases
• Backup files and databases as
objects
• Hybrid workflows for on-
premises data (e.g., data
piplelines)
• Low-latency on-premises
access to data in cloud (e.g.,
media editing)
• Content distribution and
collaboration
File Gateway Objects in your
Amazon S3 bucket
NFS v3/v4
SMB v2/v3

Volume gateway overview
• Supports iSCSI block protocol
• Available in two modes: cached and stored
• Compresses and tiers partial or whole volumesto the cloud to reduce
on-premises SAN/NAS footprint
• Take EBS snapshots of on-premisesvolumes and store them in cloud
Cloud-backed block storage presented on-premises
Key
Capabilities
Application
server
Amazon EBS
snapshots
ServiceVolume Gateway
Volume stored in
Amazon S3
HTTPSiSCSI
Customer Premises

• Emulates physical tape device with a media changer and tape drives
• Virtual tapes written to Amazon S3
• Ejected virtual tapes are marked “read only” and moved to “tape shelf” in Amazon
Glacier
• Recovery: Retrieve tapes to library (3–5 hours) and read data from a gateway
Tape gateway overview
Virtual tapes presented to on-premises backup applications
Key
Capabilities
Backup server
Tape library Tape shelf
On-Premise
iSCSI HTTPS
Media
changer
Tape Gateway
Tape
drives

Pre-installed on validated platform for predictable performance
Support
AWS
Hardware support by DELL
Easier “out-of-the-box”
Set up & manage
from the AWS Console
Hardware Appliance Updates
• Now available in Europe
• Expansion from 5 TB to 12 TB usable storage
• 10 Gb Fiber optic & copper connectivity options
Procurement
amazon.de / .co.uk/.com
Simple
Platform selection
AWS Storage Gateway on Hardware Appliance
Storage Gateway Form Factors
• VMs on VMware ESXi, Hyper-V,
& Amazon EC2
• Hardware Appliance

Storage Gateway is available on a variety of form factors
Virtual machines Amazon EC2 Hardware appliance
Corporate data centers In AWS Remote locations,
branch offices
WhereWhen
Leverage existing VM
infrastructure
In-cloud data
retrieval/access/DR
Out of the box simplified
experience

SFTP: It’s here; it’s everywhere
Protocol is deeply embedded in workflows across a variety of industries
Financial services
$
Retail
Healthcare … and more

AWS Transfer for SFTP
Fully managed service enabling transfer
of data over SFTP while stored in Amazon S3
Seamless migration
of existing workflows
Native integration
with AWS services
Simple
to use
Cost
effective
Fully managed
in AWS
Enterprise
ready

Migrate SFTP workflows to AWS
To benefit from
• Durable storage in Amazon S3
• Processing, archival and retention of data
• Integration with AWS services
SFTP SFTP server and data
Will your hundreds of partners
change their workflows to integrate
with the cloud??

In as simple as three steps
Your users can now use your AWS SFTP server endpoint to transfer data
Map your hostname Select your Amazon S3 bucket(s) Set up your users
1 2 3

Your SFTP architecture now
The same hundreds of users
now serviced by AWS Transfer,
without requiring changes
+ Enterprise ready
+ No end user disruption
+ Fully managed servers
+ Simple to use
+ Pay as you use
+ Native cloud integrations
Amazon S3AWS SFTP
AWS Transfer for
SFTP

Discuss data migration with AWS experts - STG304 - Santa Clara AWS Summit

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Discuss data migration with AWS experts - STG304 - Santa Clara AWS Summit

Ähnlich wie Discuss data migration with AWS experts - STG304 - Santa Clara AWS Summit (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

Discuss data migration with AWS experts - STG304 - Santa Clara AWS Summit