How to backup, restore and archive your data on AWS

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How to backup, restore, and
archive your data on AWS
Everett Dolgner
AWS
Chris Astall
Photobox

Agenda
Backup and archive….
AWS storage options
Getting data into AWS
Hybrid solutions
AWS Backup
CloudEndure
AWS partner solutions
Photobox: 9PB migration

Backup challenges today
• IDC predicts that the collective sum of the
world’s data will grow from 33 zettabytes
this year to a 175 ZBs by 2025, for a
CAGR of 61 percent.
• The world’s information is doubling every
two years.
• Backup and restore use cases protect data
from logical errors such as system failure,
application error, or accidental deletion.
IDC: Expect 175 zettabytes of data worldwide by 2025
Network World, Dec 3, 2018

Backup/Archive before the AWS Cloud

Backup/Archive: cloud connectors
cloud connector
HTTP/API
Cloud connector
Amazon S3 Storage
Classes
For active backup data
For long-term backup dataHTTP/API

Backup/Archive: gateways
AWS Storage
Gateway
Amazon S3
Amazon S3
Glacier/S3
Glacier Deep
Archive
Amazon EBS
Snapshots

Five key questions
Files Block Application
DB
2) Where is the data stored?
4) What is your RPO and RTO?
3) How long do you need to keep it?
5) What about the network?
1) How much data do you have?

It’s not binary.
Start somewhere
and scale up.

Complete AWS storage portfolio
Object storage
S3 Standard
S3 Glacier Deep Archive
S3 Glacier
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Standard-IA
Block storage
Provisioned IOPS SSD
Cold HDD
Throughput-Optimized HDD
NEW!
File storage
EFS Standard
EFS Infrequent Access
Elastic
Amazon EFS AWS Storage
Gateway Family
Amazon S3
Amazon FSx
for Lustre
Amazon
EBS
Amazon
EC2
NEW!
Amazon FSx
for Windows
File Server
NEW!
Backup & Restore
AWS Backup NEW!
NEW!

Amazon S3 Storage Classes
Access frequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• > 3 AZ
• $0.0210/GB
• Data with changing
access patterns
• > 3 AZ
• $0.0210 to
$0.0125/GB
• Monitoring fee per
Obj.
• Min storage duration
• Infrequently
accessed data
• > 3 AZ
• $0.0125/GB
• Retrieval fee per GB
• Min storage
duration
• Min object size
• Re-creatable, less
accessed data
• 1 AZ
• $0.0100/GB
• Min object size
• Archive data
• Select minutes or hours
• > 3 AZ
• $0.0040/GB
• Min object size
S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier
S3 Intelligent-
Tiering
S3 Glacier
Deep Archive
• Archive data
• Select hours
• > 3 AZ
• $0.00099/GB
• Min object size

Multiple Restore Speeds
Retrieval tier S3 Glacier S3 Glacier Deep Archive
Expedited 1-5 mins* N/A
Standard 3-5 hours within 12 hours
Bulk 5-12 hours within 48 hours

Cross-Region Replication
Automatically replicate data to any other AWS Region
Configure by bucket, prefix, or tag values
Replicate data to S3 Glacier Deep Archive
Change the object owner in the destination region

Lifecycle Management
Create rules to automatically transition or expire your storage
Lifecycle rules take action based on object age
Example policy:
Move all objects older than 30 days to Standard–Infrequent Access
Move all objects older than 90 days to Glacier-Deep Archive

Lifecycle Management
Create rules to automatically transition or expire your storage
Lifecycle rules take action based on object age
Example policy:
Move all objects older than 180 days to Glacier
Move all objects older than 730 days to Glacier – Deep Archive

AWS Options for Data Transfer
AWS
Direct Connect
Amazon
Kinesis
Firehose
AWS
Snowball
AWS
Snowmobile
AWS
Storage
Gateway
Amazon S3
Transfer
Acceleration
AWS
DataSync
AWS
Transfer
for SFTP
AWS
Snowball Edge
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams

AWS Snowball
AWS Snowball Edge
Compute or Storage Optimized AWS Snowmobile
• 80-TB storage capacity
• 10GE networking
• Data encryption end-to-end
• Rugged 8.5-G impact case
• Rain and dust-resistant
• 42 or 100-TB storage capacity
• Rugged 8.5-G impact case
• Rain and dust resistant
• AWS Greengrass support for local
compute, messaging, and caching
• EC2/AMI support for edge
compute
• Optional GPU
• Exabyte-scale storage in a 45-ft
container
• Dedicated security personnel
• GPS tracking, alarm monitoring,
24/7 surveillance, and optional
additional security
AWS Snow Family

Shared
file system
NFS TLS
AWS DataSync
On-Premise
Amazon S3
bucket
AWS Storage resources
AWS
DataSync
Agent deployed
on-premises for
fast access to
local storage
Region
Amazon EFS
file system
AWS DataSync
agent
Data transfer
over the WAN via
efficient purpose-
built protocol
Managed from the
console or AWS
Command Line
Interface (AWS CLI)
Service in AWS
writes or reads
data from AWS
storage services

File gateway overview
Store and access objects in Amazon S3 from file-based applications with local caching
Use cases
• Backup files and
databases as objects
• Hybrid workflows for on-
premises data (e.g., data
piplelines)
• Low-latency on-premises
access to data in cloud
(e.g., media editing)
• Content distribution and
collaboration
File Gateway Objects in your
Amazon S3 bucket
NFS v3/v4
SMB v2/v3

Volume gateway overview
• Supports iSCSI block protocol
• Available in two modes: cached and stored
• Compresses and tiers partial or whole volumes to the cloud to reduce
on-premises SAN/NAS footprint
• Take EBS snapshots of on-premises volumes and store them in cloud
Cloud-backed block storage presented on-premises
Key
Capabilities
Application
server
Amazon EBS
snapshots
ServiceVolume
Gateway
Volume stored in
Amazon S3
HTTPSiSCSI
Customer Premises
CACHE

• Emulates physical tape device with a media changer and tape drives
• Virtual tapes written to Amazon S3
• Ejected virtual tapes are marked “read only” and moved to “tape shelf” in
Amazon Glacier
• Recovery: Retrieve tapes to library (3–5 hours) and read data from a gateway
Tape gateway overview
Virtual tapes presented to on-premises backup applications
Key
Capabilities
Backup
server
Tape library Tape shelf
On-Premise
iSCSI HTTPS
Media
changer
Tape Gateway
Tape
drives
VTL

Storage Gateway is available on a variety of form factors
Virtual machines Amazon EC2
Hardware
appliance
Corporate data centers In AWS
Remote locations,
branch offices
WhereWhen
Leverage existing VM
infrastructure
In-cloud data
retrieval/access/DR
Out of the box
simplified experience

What is AWS Backup
• Centralized backup management service
• Central console and set of APIs for protecting your
application data across AWS services
• Common way to protect application data in the
AWS Cloud and on-premises
• Meet business and regulatory backup compliance
requirements
• Simple and cost-effective
AWS Backup

AWS Backup: services supported at launch
✓✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
✓
✓
Automated Backup Schedules
Automated Retention Management
Centralized Backup
Monitoring/Logging
KMS Integrated backup encryption
Lifecycle to Cold Storage
Independent Backup Encryption
Amazon
EFS
Amazon
EBS
Amazon
RDS
DynamoDB AWS Storage
Gateway

Better, Faster, More Affordable Disaster Recovery
• Improve recovery objectives & reduce TCO
• Simple setup lets you start in minutes
• Same highly automated process for all workloads
(regardless of OS type/version, application, or DBs)
• Minimizes complexity and reduces risk
• Easily failover and failback
Highly Automated
Replicate From
Any Source
Minimal Skill Set
Required to Operate
Easy, Non-
Disruptive DR Tests
ReliableFlexible
Failback to
Cloud/On-Prem
Robust, Predictable,
Non-Disruptive
Continuous Replication
Wide Range of OS,
Application, and
Database Support
Protection Against
Ransomware, Corruptions,
and Human Errors
RPO: Sub-Second
RTO: Minutes
Automated
Lightweight Staging
Area Reduces TCO

How CloudEndure Disaster Recovery Works
Lightweight Staging Area in
Target Region
Source
Data Center
Target Subnet(s)
Continuous data
replication traffic
(compressed and
encrypted)
Lightweight Linux
Replication Server
Low-Cost
Block Storage
Orchestration &
system conversion
ready to run
workloads in minutes
regardless of disk
size
• CloudEndure continuously replicates any application or database from any source into AWS
• Business outcome: Use AWS to significantly reduce DR TCO, while improving recovery objectives

Archive Partner Integrations

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The AWS Partner Network (APN) helps you seamlessly integrate AWS services into your existing storage
environment
Backup & Restore Archive Primary Storage BC/DR
AWS third-party integrations

Group history

Photobox’s journey to AWS
Started moving in 2012
The web platform was 100% on AWS by 2015
The original lift and shift evolved in a cloud native containerized
infrastructure
Now we are developing new services using AWS Serverless Technologies
Today fully in AWS

Photobox’s journey to AWS
Photo storage wasn’t immediately migrated
No significant reasons to move
Not expected to be cost effective
A migration would have been an hard technical challenge
On-Premise equipment was still fairly new and supported
However there were limitations of our on-prem infrastructure
Durability
Reliability
Both came from our single copy in single DC approach

Growth in photos
During 2018 uploads peaked at 161 millions a month and 1.1 millions in an hour

Photobox’s photo library
When we started migration
5.7 Billion individual photos / 9 Petabytes
Each of them stored in three different resolutions:
• Full-size ~ 1.6 MB
• Scaled photo < 100 KB
• Thumbnail < 10 KB
And we are growing at a rate of 1.2 B/year
Today it is 7 Billion photos and 12 Petabytes
…expected to keep increasing

Access patterns
Only 48% of photos uploaded are printed
Of those:
• 41% are printed on the day they are uploaded
• 83% within 30 days
• …and 95% within 180 days

Why moving?
Cloud first strategy
Photobox had already migrated the website into AWS and improved the operating model
Cost
About to face large capital investments for hardware expansion and replacement
Cloud storage cost kept dropping consistently
Ecosystem
AWS services can be leveraged to enhance the upload and serving workflow

Why moving? (continued)
Improved security
Access control in AWS is more flexible and granular than on-premise
Improved resilience and flexibility
Any given photo was stored in a single datacenter
Uploads were “sharded” between the two datacenters (AMS and PAR)
Storage arrays took months to scale up

How to move?
Early dilemmas: network vs Snowball Edge
With the existing 4Gbps internet uplink the migration would have taken ¾ of a year
Upgrading the production network was slow and expensive
There was no need to move everything
Scaled photos and thumbnails can be rebuilt directly into AWS
…or on the fly
Snowball Edge proved to be a good fit
82TB of useable storage per device
~110 of them needed to carry out the migration

Integrity checking
After the first use photos are hardly ever accessed again
Impossible to spot corrupted objects
The migration strategy/workflow must immediately fix any inconsistency
Hashing comes to the rescue!
In our use case the ETag of objects always matched their MD5 hash (careful: this is not always
true)
Easy to query it from S3 Inventory and compare to what has been computed at the very
beginning of the process

The migration workflow

Challenges… at scale
Network congestion
The on premise network had never been tested at 100%
Traffic flows during a migration are “atypical”
Handling an huge amount of small objects
The overhead of every single operation becomes significant
Both the data extraction and loading process had to be optimized with this in mind

Challenges… at scale (continued)
Tracking migration status
Needed to follow the status of every single object
…this results in a massive database and heavy data flow
Logistics in the physical world
In EU UPS doesn’t deliver nor collect on weekends
Next day delivery …doesn’t always happen next day

Evolution of the serving architecture
Serverless as a natural evolution
Low management overhead
Can scale instantly
Multi-region is the next big step
Keeping the live user experience consistent in case of a single region outage
S3-ZIA empowers this model

Evolution of the user journey
Amazon S3 is only the foundation
Now that the photo assets are in AWS they can be retrieved/used right away
A world of possibilities
Amazon Rekognition can be leveraged to radically change Photobox’s customer journey
Magic Book
Using AI to curating customer photos to enhance their experience

How to backup, restore and archive your data on AWS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie How to backup, restore and archive your data on AWS

Ähnlich wie How to backup, restore and archive your data on AWS (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

How to backup, restore and archive your data on AWS