With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier and AWS Storage Gateway. In addition, this session will also cover all the new AWS storage features introduced in the last 12 months.
14. Amazon S3 continuous cost reduction
• 16 Price reductions since launch
• TCO: On-premises vs. Amazon S3
– Can be challenging for some customers
– We can help!
15. RAW Storage at On perm vs. Cloud
Storage
1PB Raw Storage
800TB Usable Storage
600TB Allocated Storage
Amazon S3 Only actual
usage is
charged
400TB Written Application Storage
16. Use Amazon S3 When You Need
•
•
•
•
Unlimited storage capacity
High durability
Storage for backups
Single Origin Store with Delivery via Amazon CloudFront
23. Use Amazon Glacier When You Need
•
•
•
•
•
Inexpensive/Long-term archiving
Unlimited storage capacity
Eliminated Tape Museums
Eliminate Tech Refresh
High durability
24. Amazon S3 / Amazon Glacier Integration
POLICY-BASED ARCHIVING SERVICE
27. Archive Recovery Process with AWS &
Amazon Glacier
Glacier
S3
EC2
/HPC
CloudFront
Generating
Business
Value
$$
Hours
28.
29. Use Amazon S3 and Amazon Glacier When You Need
•
•
•
HSM in the cloud
Archive data from Amazon S3/RRS to Amazon Glacier by
policy
Delete data from Amazon Glacier by policy
42. Use Amazon EBS When You Need
•
•
•
•
•
Long-term persistent storage
Data changes frequently
Block storage for your databases – Provisioned IOPS volumes
Filesystem for an instance NTFS, ExtFS, RAID, LVM…
Access to raw, unformatted block-level storage
45. What is AWS Storage Gateway?
Integrates on-prem IT environments with Cloud storage for
departmental and remote office backup and DR
Utilizes a virtual appliance that sits in customer datacenter
Exposes compatible iSCSI interface on front end
Stores primary data on-AWS in Amazon S3 or on-premise
with data backed-up to Amazon S3 as Amazon EBS
snapshots
Amazon Confidential
45
46. Solution Overview
1. Offsite Backup
2. DR
3. Data Mirroring
AWS Storage Gateway works with your existing backup
application and moves your data into Amazon S3 as EBS
snapshots
Run AWS Storage Gateway in Amazon EC2 and access
snapshots up to 32 TB in size
Run apps in the cloud using your uploaded data –
HPC/Hadoop/Analytics
4. Dept. File Share
AWS Storage Gateway – Cached Volumes will help you
create Storage Volumes in cloud and keep most recent
accessed data locally [Reduce SAN footprint for File Shares]
5. Archive/Glacier
VTL version of AWS Storage Gateway would help customer
move data in the form of Virtual Tapes from on premise to
Amazon S3 (VTL – Virtual Tape Library) and then to Amazon
Glacier (VTS – Virtual Tape Shelf)
50. Our Story: History
iMemories was founded in 2006
Cloud was in its infancy
–
Handling physical assets:
–
–
–
Using on-premises equipment:
Focus was digitization for web/online
viewing, DVD creation, archiving
2013
•
VCRs, decks
Film projectors
Photo and slide scanners
2012
–
–
–
2010
•
VHS, Betamax, MiniDV tapes
16mm, 8mm film reels
Photos, slides
2008
•
AWS officially launched in 2006: Amazon
S3
2006
•
•
iMemories' Growth
51. Our Story: Success
1.5PB
0.75PB
2013
2012
2010
8TB
2008
•
Growing infrastructure
Reliability not as high as desired
Less-than-ideal customer experience
Tripled full-time resources to manage
Costs for static, infrequently accessed
assets much higher than for morefrequently accessed assets
Rate of growth driving need for
infrastructure elasticity
2006
•
•
•
•
•
iMemories' Growth
52. Our Story: Need
Use Case
• Video and photo
converted and digital
source files
• Short-term local copy
• Access infrequently
– Disc creation
– Additional encodings
Requirements
• Durable, redundant,
secure
• Keep forever
• No or low operational
costs
• No new hardware
• Off-premises
53. Evaluating Options: Calculating TCO
• Operations
– Percentage FTE
– Facilities and power
– Maintenance
• Hardware
– Life span
– Number of copies
– Raw GB vs. usable GB vs.
consumed GB
1.2
1
0.8
0.6
0.4
0.2
0
Time
Raw
Usable
Consumed
55. iMemories Evolution
Architecture 1.0
• All storage and processing
on-premises (circa 2006)
• Centered on physical
assets, devices, servers
• Expensive: scale by adding
more
• Non-ideal customer
experience
Architecture 2.x
• Solve scale challenges
• Reduce costs
• Best possible customer
experience
56. Storage 1.0
• On-premises
• ~1.5 PB capacity
• Limitations of scale:
– Add new hardware
– Space and power
Storage Volume
Digitization,
Encoding
Servers
iMemories Scottsdale Data center
File Mgmt
57. Storage 2.0: Amazon Glacier
Amazon SNS
Glacier
Manager
Amazon SQS
Vaults
Regions
AWS Cloud
Storage Volume
Digitization,
Encoding
Servers
File Mgmt
iMemories Scottsdale Data Center
58. Amazon Glacier
Manager
-
Interfaces with Amazon SQS
Tracks restore/submit request
state
Processes Submit requests in
real-time
Processes Restore requests
once/hour
-
-
Uses 10 MB chunks
Performs audits 3x/week
-
-
5% of consumed / 720 hours
peak hourly rate
Distribute request across 1-hr
windows
One discrepancy
Zero lost files on Glacier (!!!)
Generates internal activity reports
File Management
Glacier Manager
Digitization,
Encoding
Servers
59. 2.x: Move to AWS
v-ec2-east1.imemories.com
v-ec2-west2.imemories.com
S3 Manager
Streaming
Server
Elastic Load
Balancing
Streaming
Servers
Video
Files
Availability Zones
Regions
AWS Cloud
Storage
Volume
Digitization,
Encoding
Servers
File Mgmt
iMemories Scottsdale Data Center
60. 2.x: Move to AWS
Upload
Web App
Video
Files
Video
Processing
Group
u-ec2.imemories.com
API
Elastic Load
Balancing
File Mgmt
Amazon
SWF
Region
AWS Cloud
iMemories Scottsdale Data Center
61. In Closing
•
iMemories Story
–
–
–
•
Evaluating Storage Options
–
–
–
•
History
Success
Need
Define use case and solution requirements
Understand TCO
Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap
iMemories Evolution
–
–
–
Amazon Glacier is first full production integration
Migrating more functionality to cloud
Will always have an on-premises component
62. Enabling families to share memories in endless ways.
http://www.imemories.com
tamara.monson@imemories.com
63. Please give us your feedback on this
presentation
STG101
As a thank you, we will select prize
winners daily for completed surveys!