4. • Archiving is the process of moving data that is no longer actively used to
a separate data storage device for long-term retention. Data archives are
indexed and have search capabilities so that files and parts of files can be
easily located and retrieved.
• A backup or the process of backing up is making copies of data which
may be used to restore the original after a data loss event. The primary
purpose is to recover data after its loss, be it by data deletion or corruption.
The secondary purpose of backups is to recover data from an earlier time.
• Disaster recovery (DR) is the process, policies and procedures related to
preparing for recovery or continuation of technology infrastructure critical to
an organization after a natural or human-induced disaster.
Some Definitions
13. IT Perspective
• Engineers generate large 2D and 3D
CAD files
• CAD performance demands close
network proximity
• Drawings are Ausenco’s deliverable
and MUST be protected
14. • ASX200 company
• Head office in Brisbane
• 29 Offices in 19 countries
• 3500 Employees
Ausenco Offices
15. Our IT Environment
• Completely virtualised
• Windows
• ERP - Oracle SaaS
• Email, Lync, SharePoint – MS Office 365 SaaS
• Onsite storage
• 2 x primary data centres
• 1 x secondary data centre
16. Our Challenge
• Backup
– Unreliable
• Disaster Recovery
– Sites were exposed
• Local Disk Storage
– At capacity
Possible Solutions
An Enterprise Backup Solution?
Secondary Data Centres?
More Disk Storage?
17. Our initial approach
• Large Integrators
• Leading backup providers
• Leading archiving providers
• Hardware providers
• Data centre hosting providers
• Leading cloud providers
18. Paradigm Shift
• Why use traditional technologies?
• How can we better leverage the AWS cloud?
• How do we do more with less?
• Address the complete data lifecycle
• Flexible, Scalable, Cost Effective
Disaster Recovery ArchivingBackup
24. Business and Technical Drivers….
Reduce costs
Slash DR budgets by up to 50%
Consolidate sites
Eliminate the need to run a
secondary site
Reduce on-premise
Eliminate 30%+ of on-premise
physical equipment
Remove aging
technologies
Eliminate tape for backup and
archive
25. The fundamental economic model…
Utility, on-demand datacenter
Primary Site
Routers
Firewalls
Network
Application Licenses
Operating Systems
Hypervisor
Servers
SAN
Primary Storage
Backup
Archive
AWS
Routers
Firewalls
Network
Application Licenses
Operating Systems
Hypervisor
Servers
SAN
Snapshot Storage
Backup
Archive
Secondary
site costs
27. Backup Lessons – My backup should be accessible
Source: http://www.abc.net.au/news/specials/qld-floods/
a.k.a. the pain
of physical
data transfer
28. AWS Direct Connect
Dedicated bandwidth between
your site and AWS
Amazon Storage Gateway
Shrink-wrapped gateway for volume
synchronization
AWS Import/Export
Physical transfer of media into and
out of AWS
Getting data into the cloud
29. Simple Storage Service
Highly scalable object storage
1 byte to 5TB in size
99.999999999% durability
Elastic Block Store
High performance block storage device
1GB to 1TB in size
Mount as drives to instances with
snapshot/cloning functionalities
Glacier
Long term object archive
Extremely low cost per gigabyte
99.999999999% durability
Storage Options
Very fast
‘instance’ disks
Slow, rare accessFast web object
storage
35. • “Infinite” scale with Amazon S3 and Amazon Glacier
• Scale to multiple regions
• Seamless
• No need to provision
• Cost tiers (cheaper at scale)
Backup Lessons – My backup should be able to scale
36. • SSL Endpoints (Amazon S3 and Amazon Glacier)
• Signed API calls
• Store encrypted files
• Server-side encryption
• Multiple copies across different data centers
• Local/cloud with AWS Storage Gateway
Backup Lessons – My backup should be safe
37. • Easy to integrate within AWS or Hybrid
• AWS Storage Gateway: Run services on Amazon EC2 (DR)
• Clear costs
• Reduced costs
• I decide redundancy/availability in relation to costs
Backup Lessons – My backup should work with a DR policy
39. • Clear ownership
• Permissions with IAM: Users, groups roles
• Logs
• AWS support
Backup Lessons – Someone should care about it
40. 1. My backup should be accessible
1. My backup should be able to scale
1. My backup should be safe
2. My backup should work with a DR policy
3. Someone should care about it
Backup Lessons
42. DR is part of a wider set of policies and controls…
DR & business continuity
It’s not an all or nothing thing
Choose what needs to failover and what does not
Some things more important than others
Some things will still be working
High availability Backup Disaster recovery
Keep your applications
running 24x7
Make sure your data is safe Get your applications and
data back after a major
disaster
43. Each set of IT assets will have different requirements…
DR & business continuity
Recovery Time
Objective (RTO)
How quickly you need this asset to be
recovered?
e.g. 1min? 15min? 1hr? 4hrs? 1day?
Recovery Point
Objective (RPO)
How ‘fresh’ the recovery must be for the
asset?
e.g. zero data loss, 15mins out of date?
44. Assets will sit on a spectrum of technical complexity…
DR & business continuity
Rebuild when
required from
offsite backup
Run hot-hot
configuration with
auto-failover
45. DR Lessons – You NEED a DR plan in place
DR with High Availability
46. App DR with Standby
DR Lessons – You NEED a DR plan in place
47. DR Lessons – Testing your DR
• Dev/test in the cloud is super easy
• Spin up capacity only for the test
• Regularly test your DR
• Cost is minimal
• What about data transfer speed?
s3cmd ls --recursive
s3://datasets.elasticmapreduce/ngra
ms/books/ | awk '{print $4;
sub(/s3://datasets.elasticmapredu
ce/, "/array", $4); print $4}' |
parallel -j0 -N2 --progress
/usr/bin/s3cmd --no-progress get
{1} {2}
Copying 2.4 TB
down from 48 hours
to 9 hours (5x
faster)
48. DR Lessons – Reducing Costs
• Dev/test in the cloud is super easy
• Spin up capacity only for the test
• Regularly test your DR
• Cost is minimal
• What about data transfer speed?
49. DR Lessons – You can have different DR solutions
• Easy to integrate existing vendors with DR on AWS
• Approach: One vendor/hybrid/multiple vendors
• One region/multi-regions (if you need geo-diversity)
• Different DR Architectures
Backup & Restore Pilot light
Warm standby in
AWS
Multi-site solution
in AWS & on-
premise