AWS Update | London - Amazon Glacier

Amazon Web Services Update | London
November 2012

Getting to Glacier…
Why AWS for storage & archive?
AWS fundamental services
Storage & archive – examples &
patterns
Amazon Glacier

Storage & Archive
AWS is used in a variety of ways…

Powers applications that allows
customers to access historical Store its vast repository of music to
stock price information feed to over 15 million active users

Estimates it has saved $500,000 Digital assets and usage data behind
in storage expenditures and cut publication sites and mobile
its disk storage array costs in half applications

Business & technical drivers
You might be able to:

Reduce costs Reduce on-premise
Slash storage & archive budgets Eliminate on premise equipment to
manage archives

Change processes Remove aging technologies
Remove the need to do capacity Eliminate tape for backup and archive
planning


Reduce CAPEX while dramatically
Slash storage & archive budgets by Eliminate on premise equipment to
increasing scalability
up to 50% manage archives
Eliminate the need for secondary
sites

planning


Reduce CAPEX while dramatically Eliminate 30%+ of your storage
increasing scalability footprint
Eliminate the need for secondary Consolidate on-premise and
sites augment with cloud

planning



Remove the need to do capacity
Eliminate capacity planning Eliminate tape for backup and archive
planning
Eliminate provisioning for peak
demand



Remove the need to do capacity
Eliminate capacity planning Eliminate tape for backup and
planning Remove tape archives
Eliminate provisioning for peak
Cycle out aging disk arrays
demand

Fundamental Storage Options
Elastic Block Store, S3 and Glacier

Elastic Block Store Simple Storage Service Glacier
High performance block storage device Highly scalable object storage Long term object archive
1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte
Mount as drives to instances with 99.999999999% durability 99.999999999% durability
snapshot/cloning functionalities


Elastic Block Store Simple Storage Service Glacier
1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte
Mount as drives to instances with 99.999999999% durability 99.999999999% durability
snapshot/cloning functionalities

Very fast Fast web object Slow, rare access
‘instance’ disks storage


Elastic Block Store
Archive Backup
Simple Storage Service
DR
Glacier
Data1TB in size
1GB to accessed Snapshots 1 byte to 5TB in size Extremely low cost per gigabyte
Rapid RTO
~>10% / month
Amazo as drives to instances with
Mount 99.999999999% durability 99.999999999% durability
Shorter term data
nsnapshot/cloning functionalities backup with rapid
S3 Expiration policies
11 9s durability
RTO

Amazo
Lower cost when 11
n S3 9s not required
Lower cost Lower cost
RRS

Long term
Amazo archiving Use policies to Retain “write once -
move cold backup read never” copy in
n Infrequent data data for long term case of worst case
Glacier access (~<10% retention scenario
data/month)

Use case journey
On-premise On-instance Object level Long term

Locally
accessible file
systems
Workloads
with local data

Use case journey

Locally
accessible file
systems AWS
Workloads
with local data

Use case journey

Locally EC2 based Data System images
accessible file applications distribution Database
systems DR Durable media backups
Workloads deployments storage Data archives
with local data

Use case journey

Locally EC2 based Data System images
with local data

High IO High IO performance Good Very low price
performance Provisioned IOPS performance High durability
High network Backup & Restore High durability Slow access
performance Scalability

Use case journey

Locally 1 EC2 based Data System images
with local data
Getting
data into
the cloud

Getting data into the cloud
Direct connect, import/export and storage gateway

AWS Direct Connect AWS Import/Export Amazon Storage Gateway
Dedicated bandwidth between you Physical transfer of media into and Shrink-wrapped gateway for volume
site and AWS out of AWS synchronization

Use case journey

accessible file applications and distribution
Disks
Database
systems data
DR Durable media backups
with local data
Getting
data into 2
the cloud

Curiosity
The mars.jpl.nasa.gov website
is based on the open-source
Content Management System
(CMS) Railo, running on
Amazon EC2

Shared storage for Railo is
provided by Amazon EC2
instances running Gluster on a
pool of Amazon Elastic Block
Store (EBS) volumes for
consistently high performance
disk I/O.

Use case journey

Disks
Database
systems data
with local data
Getting
data into 2
the cloud
Database
performance
as a
service
3 Scalability

Database services
RDS, SimpleDB, DynamoDB

Relational Database SimpleDB DynamoDB
Schemaless NoSQL, Schemaless,
Service Smaller datasets Provisioned throughput
Fully managed database
database
(MySQL, Oracle, MSSQL)

Use case journey

Disks
Database
systems data
with local data
Getting
data into 2 4
the cloud
Database
performance
Object
as a
service
3 Scalability
serving
and
storage

You put in it S3
AWS stores with 99.999999999% durability

Highly scalable web
access to objects

You put in it S3
AWS stores with 99.999999999% durability

Multiple redundant
copies in a region

“Spotify needed a storage solution that
could scale very quickly without incurring
long lead times for upgrades. This led us to
cloud storage, and in that market, Amazon
Simple Storage Service (Amazon S3) is the
most mature large-scale product.

Amazon S3 gives us confidence in our
ability to expand storage quickly while also
providing high data durability.”
Emil Fredriksson, Operations Director

Use case journey

Locally 1 EC2 based Data Cold System images
accessible file applications and distribution storage & Database
Disks
systems data
DR Durable mediaarchiving backups
with local data
Getting
data into 2 4 5
the cloud
Database
performance
Object
as a
service
3 Scalability
serving
and
storage

What we heard from you

You love Amazon S3
for its simplicity,
security, durability,
and performance.

What we heard from you

You love Amazon S3 You wanted a highly
for its simplicity, secure, extremely
security, durability, durable, and extremely
and performance. cost effective option for
archiving data for years

The need…

Reliable and cheap storage of data

Data with long Multi-PB, infrequently
retention periods accessed data sets

Our goals with Glacier…

Redefine data archiving Replace physical media for
and backup: archiving:
no upfront payments an easy to use storage service that is
a very low price for storage infinitely scalable
ability to scale up and down as a secure service for important data
needed assets
designed for an annual average
99.999999999% durability per saved
object

for as little as $0.01 per gigabyte per month

The solution…


The solution…


Same storage
durability mechanisms
as S3

The solution…


Same storage Trade-off on retrieval
durability mechanisms time
as S3

3-5 hour retrieval time
We assume you won’t access often

Glacier allows you to cost-effectively and securely store
Offsite archive enterprise data offsite, making it simple, inexpensive and safe
to retain archived data for as long as desired. Common use
cases include enterprise data, media assets, and research and
scientific data

scientific data

Libraries, historical societies, non-profit organizations and
Digital preservation governments are increasing their efforts to preserve
valuable but aging digital content such as websites, software
source code, video games, user-generated content and
other digital artifacts

scientific data

Libraries, historical societies, non-profit organizations and
Digital preservation governments are increasing their efforts to preserve
valuable but aging digital content such as websites, software
source code, video games, user-generated content and
other digital artifacts

Amazon Glacier is cost competitive, even at scale, and
Tape replacement eliminates pain points like capacity planning, capital
budgeting and investments, media formats, hardware
refreshes, and off-site storage costs, shipping and
retrieving

Good reasons to replace off-site tape archives

100% restore success rate – no broken or missing
tapes
No lost tapes and improved security posture
No device or media admin or handling
No capacity planning
Pay as you go
No need for recurrent and risky data migrations

S3 Glacier

Bucket Vault

Object Archive

Create vault supported via console

What is an archive?
Any object, such as a photo, video, document or
compressed collection
It is a base unit of storage in Amazon Glacier
Upload an archive in a single request
For large archives use multipart upload API

API credentials
Glacier client (keys)
Region endpoint

client = new AmazonGlacierClient(credentials);

client.setEndpoint("https://glacier.us-east-1.amazonaws.com/");

ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials);

UploadResult result = atm.upload(vaultName, ”MyArc “, new
File(archiveToUpload));

Transfer manager
File to upload
Vault & archive
name Java

Transfer manager
Region endpoint

var manager = new
ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);

string archiveId = manager.Upload(vaultName, ”MyArchive",
archiveToUpload).ArchiveId;

Vault & archive
File to upload name
.net

Retrieval

S3 Glacier

Synchronous Asynchronous

Immediate 3-5 hours

Retrieval

1. Initiate a retrieval job
2. After the job completes, download the
bytes

Initiate job

JobParameters jobParameters = new JobParameters()
Glacier .withArchiveId("*** provide an archive id ***")
client .withDescription("archive retrieval")
.withType("archive-retrieval");

InitiateJobResult initiateJobResult =
client.initiateJob(new InitiateJobRequest()
.withJobParameters(jobParameters)
.withVaultName(vaultName));

String jobId = initiateJobResult.getJobId();

JobID to track Java

Track job

After 3-5 hours:
1. SNS topic notification
2. Call describeJob
Using JobID

API credentials Download job
Glacier client (keys)
Region endpoint

client = new AmazonGlacierClient(credentials);

client.setEndpoint("https://glacier.us-east-1.amazonaws.com/");

ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials);

atm.download(vaultName, archiveId, new File(downloadFilePath));

Transfer manager
Download path
Vault name &
archive id Java

Download job
var manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);

var options = new DownloadOptions();
options.StreamTransferProgress += ArchiveDownloadHighLevel.progress;

manager.Download(vaultName, archiveId, downloadFilePath, options);

static int currentPercentage = -1;
static void progress(object sender, StreamTransferProgressArgs args)
{
if (args.PercentDone != currentPercentage)
{
currentPercentage = args.PercentDone;
Console.WriteLine("Downloaded {0}%", args.PercentDone);
}
}

.net

“Every day our genome sequencers produce
terabytes of data. As our company moves into
the clinical space, we face a legal
requirement to archive patient data for years
that would drastically raise the cost of
storage.

Thanks to Amazon Glacier’s secure and
scalable solution, we will be able to provide
cost-effective, long-term storage and thereby
eliminate a barrier to providing whole genome
sequencing for medical treatment of cancer
and other genetic diseases.”

“An organization like ours thinks in centuries
when it comes to content retention, and long
term preservation of our Master Archives is a
critical part our mission here at NYPR.

Storing these core assets on traditional media
such as local disk and off-site tape exposes us to
corruption and even outright-loss of data. We
are excited to move our archives to Amazon
Glacier, which will be a better long-term
solution.”
Steve Shultis, CTO, New York Public Radio

Desktop clients

S3 integration coming soon

Use Glacier through S3 APIs
Policy based tiered storage

Storage Retrievals Data In Data Out
From $0.1 per GB Free up to 5% of Free Tiered (1st GB free)
average monthly
storage the tiered
fees

Storage Retrievals Data In Data Out
From $0.1 per GB Free up to 5% of Free Tiered (1st GB free)
average monthly
storage the tiered
fees

Anticipation is archives will be accessed infrequently
Storage is cheap, trade-off on retrieval pricing

Benefits of Amazon Glacier
Low cost Secure

As little as $0.01/GB/month with no up-front capital Secure and durable technology platform with
commitments. industry-recognized certifications and audits.

Durable Simple

Average annual durability of 99.999999999% per Eliminate hardware, software, and capacity
archive. planning.

Flexible Use multiple services

Add any amount of data, quickly. Easily expire and Easily leverage other AWS services once your data is
delete without handling media. in the AWS cloud.

http://aws.amazon.com/gla
cier/

AWS Update | London - Amazon Glacier

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie AWS Update | London - Amazon Glacier

Ähnlich wie AWS Update | London - Amazon Glacier (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

AWS Update | London - Amazon Glacier