The document discusses AWS Snowball, Snowball Edge, and Snowmobile - physical data transport solutions for migrating large amounts of data into AWS. Snowball is designed for petabyte-scale data migration, Snowball Edge provides petabyte-scale hybrid storage and compute capabilities, and Snowmobile is for exabyte-scale data migration using a 45-foot shipping container. The document provides details on their capabilities, use cases, security features, and cost. It also includes examples like how Oregon State University uses Snowball to migrate terabytes of oceanic research data and how DigitalGlobe migrated 100 petabytes of satellite imagery to AWS using Snowmobile.
2. Data Migration, Batch and Stream Processing
Direct
Connect
Snowball,
Snowball Edge,
Snowmobile
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
3. Why are we here?
Large Customer:
“We want to move 250 PBs into AWS, but it would
take years to do that over the network.
How can we upload our data faster into AWS?”
4. Why are we here?
Data is growing at a tremendous pace!
4k/8k
Genomics
Seismic
Financial
Logs
IoT
100TB over the Internet: 2 years
1PB over the internet: 22 years
100PB over T3: 609 years
5. AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale
data migration
6. AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale
data migration
7. The AWS data transport solution
AWS Snowball—petabyte scale data transport
• Rugged 8.5G impact case
• Rain and dust resistant
• Data encryption end-to-end
• 50 TB capacity/10G network
• E-ink label
E-ink shipping
label
RE:INVENT 2015 LAUNCH
8. Snowball features released post re:Invent 2015
• Data export
• 80 TB Snowball
• Snowball SDK
• S3 Adaptor
• HDFS import
• HIPAA eligibility
9. How is my data transported securely?
• 256-bit encryption
• Data is encrypted by the
Snowball client before it is
written to the Snowball
appliance
• Keys are managed by AWS and
are never written to the Snowball
10. How is my data transported securely?
• Strong chain of custody
• Tamper-resistant case
• Tamper-resistant electronics
(TPM)
• Each Snowball is erased
according to NIST 800-88 media
sanitization guidelines between
every job
11. How Snowball moves data into and out of AWS
Create
a job
Connect the
Snowball
Copy data to
the Snowball
Your data
moved to
Amazon S3
In transit to you Delivered to you Delivered to AWS At AWS
Job created Job completed
13. When to use AWS Snowball
Cloud
Migration
Disaster
Recovery
Datacenter
Decommission
Content
Distribution
14. Customer use case: S3 as a data lake
Amazon EMR Amazon Kinesis
Amazon
Redshift
Amazon
DynamoDB
Amazon RDS
AWS Data Pipeline
Spark StreamingCassandra Storm
Amazon
S3
15. Case Study: Oregon State University
Use-case:
• Collect and analyze oceanic and coastal images
• 60TB of data per week
• Environmental and ocean ecosystem research
Architecture before Snowball:
• Transferred data with many small hard drives
• Used to take weeks to months to upload data
• $4MM+ in infrastructure investment
• Expensive and inefficient
Snowball lets OSU migrate TBs of data in days at a fraction of the cost
16. What does Snowball cost?
• Per Job shipping
- $250 / Job plus shipping (80TB)
- $200 / Job plus shipping (50TB)
• Includes 10 days to fill the device at your site
• $15/day after the tenth day on site
• Standard S3 charges apply
• $0.00/GB to transfer data in
17. AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale
data migration
18. AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equivalent to an
Amazon EC2 m4.4xlarge instance
• 10GBase-T, 10/25Gb SFP28, and
40Gb QSFP+ networking
• Ruggedized and rack-mountable
RE:INVENT 2016 LAUNCH
19. Snowball Edge key features
S3-compatible endpoint
File interface (NFS)
Clustering
Run AWS Lambda functions
Faster data transfer
Encryption
20. Hybrid capabilities beyond data migrationMIGRATION
Create job Copy data, Ship to AWS Moved to S3
Collect & Process data
PROCESSING
Create job Ship to AWS Moved to S3
21. Snowball Edge can…
Extend of your
data center
Process data Expedite
move
Encrypted, secure,
and embedded
compute
Write data directly
as data is generated
Offers a fast and
cost effective way to
ensure data can be
quickly transferred to
and from the cloud
Simplify
data transfer
Uses standard
and familiar tools
for the data
transfer process
22. Snowball Edge applicability
• Incorporate Snowball Edge into existing applications, i.e.
• Backup destination
• File/object store
• Data cluster
• …
• Build new, end to end applications with Snowball Edge
• In the manufacturing plant
• In the hospital
• In automobiles
• …
23. Snowball Edge use cases
Offline
Staging
Local Tiering
and Compute
IoT
Local
Transformation
24. Aviation
•Use Cases:
- Planes equipped with cameras
- Fly over regions to build 3D models
- Run algorithms on content for data mining
•Current solution:
- Cameras stream data to massive HDDs which are shipped to DC
•How Snowball helps?
- Migrate directly to AWS
•How Snowball Edge helps?
- Identify operational failures in trips for immediate action
25. Healthcare Industry
Use Cases:
- Typical hospital's data repository can exceed 100s of TBs
- Expensive (time and $$$) to transfer data to AWS
- How to integrate with solutions like Philips HeathSuite to provide
healthcare professionals with data quickly
How Snowball helps?
- Can transfer data in days instead of weeks
- Can easily scale to hundreds of hospitals
- Meets HIPAA requirements
How Snowball Edge helps?
- While data is being stored, can we detect anomalies to instantly
raise alerts?
26. Oil & Gas
Ben Wilson (CTO of GE Oil and Gas):
“PIGs and Snowballs—a match made in heaven! AWS
Snowball 25 TB of pipeline PIG data to be managed at
AWS. That is our GE PIG we pulled some of the data
from. It’s always fun to try new AWS features and try to
break them!”
•Use Cases:
- PIG is a device to inspect cracks in pipelines using sensors
- Device fits inside pipeline and sent through to measure quality
•How Snowball helps?
- Use Snowball to migrate data to AWS
27. Security
• Use Cases:
- Biometric system uses face detection for security systems
- Replacing security guards with automated security systems
- Dozens of video cameras recording 8-10 hours/day
- Tens of thousands of videos a week
• How Snowball helps?
- When video is recorded, goes to local storage and then shipped to AWS
• How Snowball Edge helps?
- Do lot of image processing; Can use Lambda to do on premise
processing
29. Getting started with Snowball Edge
Step 1: Create a Job and wait for Snowball to ship to you
30. Getting started with Snowball Edge
Step 2: Power it on and connect to network
Step 3: Install the Snowball Edge Client Software
Step 4: Note the IP address on the Kindle Fire of device
Step 5: Get the Manifest file and Unlock code from AWS
Console
31. Getting started with Snowball Edge
Step 6: Unlock the device
snowballEdge unlock -i 192.168.1.141 -m C:sfloft.bin -u 906d9-
960bf-3e3f2-99aa2-79bf1
Step 7: Check the status of the Snowball
snowballEdge status -i 192.168.1.141 -m C:sfloft.bin -u 906d9-
960bf-3e3f2-99aa2-79bf1
Step 8: Get the S3 credentials
snowballEdge credentials -i 192.168.1.141 -m C:sfloft.bin -u
906d9-960bf-3e3f2-99aa2-79bf1
32. S3 operations
You can use AWS CLI for S3 commands:
• Store the S3 credentials under .aws/credentials
• aws s3 ls s3://rashimgcmh --endpoint http://192.168.1.141:8080
• aws s3 cp c:bar.txt s3://rashimgcmh --endpoint
http://192.168.1.141:8080
35. Solve Seeding Headaches with
NetApp ® AltaVaultTM
and AWS Snowball
Primary
Storage
Cloud-Integrated
Storage
AltaVault™
3rd Party
Backup and
Recovery
Solution
Encryption
De-duplication
Compression
NetApp® AltaVault
TM
AWS Snowball
36. What does Snowball Edge cost?
• Per Job shipping
- $300 / Job plus shipping
• Includes 10 days to fill the device at your site
• $30/day after the tenth day on site
• Standard S3 charges apply
• $0.00/GB to transfer data in
37. AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale
data migration
39. What
• 10-100PB in a 45 foot-long ruggedized container truck
Where & When
• Can be made available in all AWS regions
How
• Data transferred via multiple 40Gbps interfaces up to 1Tb/s (100PB in a few weeks)
• Appears as HTTP end point
• End-to-end secure (256-bit encryption)
• Customer orders a Snowmobile, we dispatch it to their site, they hook it up and fill it, it returns
How much does it cost
• $0.005/GB/mo based on provisioned capacity (from site departure to AWS ingestion completion)
40. Snowmobile key features
Massively Scalable
Customized for your needs
Faster data transfer
Strong Encryption
Rugged, Durable, and Secure
41. Snowmobile Case Study: DigitalGlobe
Use-case: Seeing a better world
• DigitalGlobe takes Satellite imagery of the Earth
• 100PB image library = 6 billion square kilometers
• 1PB new image every year
Architecture before Snowmobile:
• Stored data in their own data center
• Needed elastic compute power to retrieve and analyze images
• Wanted to move data to the cloud, but no feasible solution
Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud
42. Summary
Typical scenario Data Migration Data Migration or
Local compute
Data Migration
Storage 50TB, 80TB 100TB 100PB
Compute N/A Comparable to m4.4xlarge N/A
AWS Services available S3 Lambda, File Gateway, S3 N/A
NFS Endpoint N/A Yes Yes
Encryption Yes Yes Yes
Clustering for local use N/A Available N/A
Typical job lifetime Days - weeks Data Migration – Days-Weeks
Local compute – Weeks - Years
Weeks – months
HIPAA Yes Roadmap No
Snowball SnowmobileSnowball Edge
Problem Statement: How do you get your data from your primary storage and into AWS quickly, efficiently, and securely? AltaVault and Snowball
AltaVault will allow you to use your existing backup applications (24 backup applications certified to work with AltaVault)
Typical 6.5x reduction via deduplication and compression (but up to 30X reduction)
Resulting in 325TB of raw data on a 50TB Snowball (but up 1.5PB)
Resulting in 520TB of raw data on a 80TB Snowball (but up to 2.4PB )
Data is encrypted upon AltaVault ingest, through transport, and while at rest in AWS
After upload, AltaVault will sync with the customer’s AWS bucket
Recoveries are available immediately after Snowball data is uploaded by AWS into customer’s bucket
Subsequent backup jobs sent to the AltaVault are automatically sent to AWS over the Internet and added to the data already uploaded by the Snowball