Paul Angus – Backup & Recovery in CloudStack

The Cloud Specialists
CloudStack Backup and
Recovery Framework
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus
paul.angus@shapeblue.com
A P r e vi e w o f

ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public
& private clouds. They are the leading
independent global CloudStack services
company”
Ab o u t S h a p e B l u e

C l i c k t o e d i t
Paul Angus – VP Technology
• Global authority on CloudStack & cloud infrastructure design.
• 20+ years C-Level experience.
• Apache CloudStack project committer & PMC member
• Specialising in deployment of CloudStack
and surrounding infrastructure especially the user story
• USP, Georgian Ministry of Justice, Orange, TomTom,
PaddyPower, Ascenty, BSkyB, SAP, British Telecom
Ab ou t m e

The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

• What is a snapshot? (and what’s snapshots got to do with this
anyway?)
• Why is this a ‘framework’? (and why is it in quotes?)
• What does this ‘framework’ do then? (enough with the quotes)
• What will the framework work with?
• An Example would be nice ..
Cl ou d S t ac k B ac k u p an d Rec over y F r am ewor k

What’s is a Snapshot?

VM Snapshot
• A hypervisor-driven point-in-time image of a virtual machine’s
disks. The exact mechanism of this is dependant on the
hypervisor.
Volume Snapshot
• A point-in-time image of a specific volume – The process usually
involves taking a VM snapshot and then copying the required
volume to secondary storage and the deleting the VM snapshot
W h at ’ s a S n ap s h ot ?

Features of volume snapshots include:
• Alternate storage location of image (from primary storage)
• Ability to schedule the volume snapshots
• Ability to set a retention policy of n copies.
• Ability to create new VMs from the snapshot image (pseudo
restoration)
• Ability to export volume snapshots from the system
O K , b u t wh at ’ s s n ap s h ot s g ot t o d o wi t h t h i s ?

So snapshots are used by many (everyone) as a form of
backup
but…
O K , b u t wh at ’ s s n ap s h ot s g ot t o d o wi t h t h i s ?

What’s wrong with
snapshots as backups?

W h a t ’ s w r o n g w i t h s n a p s h o t s ?
• Not consistent across volumes
• A VM with separate disks for say; OS, logs and
data, will have each disk processed at different
times
• Only crash consistent within a volume
and they aren’t application aware
• A database will not be quiesced before snapshot is
taken
• Transaction logs won’t be processed
• VMware tools or XenTools may help with some
basics

• They can be very slow.
• Data has to be transferred from primary storage to
secondary storage
• In the case of vSphere, this data must go via the
SSVM to be compressed into a OVA while being
moved to secondary storage.
• CloudStack locks out other actions while a
snapshot is taking place. Workarounds have their
own issues.

• Restoration of a VM requires a number of steps
(and did I mention it’s slow)
• User must make a template from snapshot
(VM image copied via SSVM from one part of secondary
storage to another – 2x network transfers)
• User then creates VM from template
(VM image copied from secondary to primary to create
template on primary storage, then primary storage based
template copied to make actual VM. 2x network assuming
network based primary storage and 1x on-disk copy)
• To restore a VM ‘exactly as it was’ requires considerable
operator intervention

Why is this a ‘framework’

The CloudStack Backup & Recovery Framework
• This is a framework because, by itself it doesn’t ‘do’ anything.
• Plugins are required to drive physical actions
• A dummy plugin has been created for testing the B&R framework API
Oh. This is going to be a short/rubbish presentation then.
W h y i s t h i s a ‘ f r a m e w o r k ’ ?

What does this ‘framework’
do then?

The CloudStack Backup and Recovery Framework
Creates a vendor agnostic API and UI in CloudStack for end users to leverage 3rd
party backup and recovery solutions.
The B&R Framework abstracts the specifics of B&R solutions, such that through the
use of a plugin, a 3rd party B&R solution can deliver ….
W h at d o e s t h i s ‘ f r am ewor k ’ d o t h en ?

For the Cloud Operator:
The B&R framework enables cloud operators to leverage their choice of enterprise
grade backup and recovery technology. Giving them:
• Greater control over the backend backup and recovery processes
• Hardware and software accelerated performance
• Reducing/elimination of operator overhead
• More features to offer users
• Additional chargeable services

For the Cloud Consumer:
• Improved performance over snapshots
• A feature set that consumers expect from backup software
• Proper restoration options
• Consistent API and user experience independent of B&R vendor
• Piece of mind of a recognised industry named B&R solution

Specifically:
• Support for:
• Operator created ‘policy based’ backups (RPOs)
• Scheduled backups (similar to snapshot scheduling)
• Ad hoc backups
• Backups as first class citizens
• ‘In-place’ restoration of entire VM including data disks
• Restoration of VM backup after original VM is destroyed
• Restoration of individual VM volumes (can then be attached to another VM)
• Full integration into the CloudStack usage service, recording both protected and
on-disk data metrics

What will the framework
work with?

Dummy Plugin
• Built for compile time testing of the APIs.
Veeam Backup & Replication 9.5 (for VMware)
• Plugin written against Veeam Backup & Replication 9.5 and Apache CloudStack
4.12
• Tested against VMware 6.5
• Requires Veeam Backup & Replication Console, Veeam Backup Enterprise
Manager, OpenSSH and PowerShell on the Veeam Server
W h at w i l l t h e f r am ew or k w or k w i t h ?

Example(s) ?

Backup APIs so far*
• createVMBackup(A)
• deleteBackupPolicydeleteVM
• BackupimportBackupPolicy (A)
• listBackupPolicies
• listBackupProviders
• listVMBackupRestorePoints
• listVMBackupsrestoreVMBackup (A)
• restoreVolumeFromBackup (A)
• startVMBackup (A)
*Still subject to change during development
Th e B a c k u p an d Rec over y AP I

Veeam B & R P l u g i n
‘Policy Based’ Backups
• listBackupPolices external=true
Returns ‘template’ jobs.
• importBackupPolicy Creates a policy (with
a name and description) in CloudStack
and maps it to a Veeam template job.
ie.
GOLD – 6hr RPO
SILVER – 24hr RPO
BRONZE – 7 day RPO

‘Policy Based’ Backups
• Add VM to the policy using
createVMBackup*
Creates a Veeam backup job for the VM
with the properties of the template job
linked.

‘Ad-hoc’ & Scheduled Backups
• CloudStack will send command to carry
out an ad-hoc run of the policy based job
or create one as required.
• Ad-hoc and scheduled jobs will be driven
by CloudStack
• One VM to one Veeam job mapping allows
for simpler accounting and unified
incremental backups

Listing Backups
listVMBackups
• Lists backup jobs for a given VM. Under
Veeam approach, this is a one-to-one
relationship
listVMBackupRestorePoints*
• Veeam thinks in terms of restore points as
being the image of a VM at any point in
time. This API lists the restore points
available for a given VM backup job (VM)
*This is quite Veeam centric and potentially confusing, as the
overall feature is rounded out, the abstraction naming will be
reviewed.

VM and Volume Restoration
• Veeam ‘in-place’ restoration is more in-place-of than in-the-original-place
• The Veeam plugin takes care of the fact the replacement VM is in an alternate
location to the ‘original’
• Current implementation automatically attaches a restored volume to a VM (option
to restore to secondary storage will be added)

That’s all cool. Where is it?

Framework & Veeam Plugin
• Expect to see a PR in Q4
• APIs largely done
• Work left:
• Scheduling of backups
• Ad-hoc backups
• Changing of underlying policy
• Deletion of backups/restore points
UI
• Expect to see a PR late Q4
Ti m e l i n e

Questions?

Paul Angus – Backup & Recovery in CloudStack

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Paul Angus – Backup & Recovery in CloudStack

Similar to Paul Angus – Backup & Recovery in CloudStack (20)

More from ShapeBlue

More from ShapeBlue (20)

Recently uploaded

Recently uploaded (20)

Paul Angus – Backup & Recovery in CloudStack