Linux Disaster Recovery Made Easy

Linux Disaster Recovery
Made Easy

Schlomo Schapiro Jeff Lindholm
Systems Architect / Open Source Evangelist Technology Sales Specialist – Linux
Immobilien Scout GmbH Novell, Inc.
schlomo.schapiro@immobilienscout24.de jlindholm@novell.com

Agenda

Linux and disaster recovery

Solutions: relax and recover

Secure disaster recovery for enterprise environments

SUSE Linux Enterprise Server 11 SP1 HA –
®

ReaR availability and support

Live demo and use cases

Questions and answers

2 © Novell, Inc. All rights reserved.

Disaster Recovery Strategies

Copy Files and Store Disk Disk Imaging
Layout and Boot Info

+ Online -> no problem + Offline -> no open
+ Backup independent of disk file issues
layout and sizes + Simple to restore
– More effort required to – Online -> very
restore (can be scripted) problematic
– Maybe consistency – No “perfect” open
problems, but should be source tool for
solved by backup solution Linux available


Advantages of a Linux System

All information is stored in files; all files always readable

Operation system and applications can be “slim” -
600MB enough for complete standard server

Open system – open methods and procedures

All steps of an installation can be scripts:
partitioning, file systems, boot loader etc.


Disaster Recovery – Media

• Most important: external storage!
• Bootable media: CD/DVD, USB key, LAN, tape ...
• Media usually combination boot and backup media:
– Bootable CD/DVD, USB key with backup data on it
– LAN boot (PXE) with backup data via CIFS, NFS ...
– Bootable tapes - HP OBDR (CD emulation)
• Separation between boot media and backup data
– Boot the system from a (small) USB key, CD/DVD or LAN
– Recover the system with backup software, tar, rsync ...


Disaster Recovery – How It Works

• Store the disk layout
– Partitioning, LVM and RAID configuration
– File systems, file system labels ...
– Boot loader (GRUB, LILO, ELILO)
• Store the files (tgz, rsync, through backup software ...)
• Create bootable rescue media with system
configuration (and backup data)
• Can be done online
– No business interruption
– 100% compatible with original systems hard- and software

Disaster Recovery – Rescue Media

• Create “rescue linux” from running system
• Optimally compatible “tool box”
• Clone the system environment
– Linux kernel and modules

– Device driver configuration

– Network configuration

– Basic system software and tools

• Operate entirely in RAM (initrd)


Disaster Recovery – In Action

• Boot system from rescue media
• Restore disk layout
– Create partitions, RAID configuration and LVM
– Create file systems (mkfs, mkswap)
– Configure file systems (labels, mount points)
• Restore the backup data
• Restore the boot loader
• Reboot
• Done!

Commercial Solutions

• Usually part of backup software
• Extra licenses and can be costly (up to 1000€ / server)
• Some are free, but are the any good?
• Level of integration into backup software varies
• Not all backup software actually supports
disaster recovery
• Cover only simple setups
• Often static boot media, inflexible with regard to drivers
• Certification on server hardware missing or lagging
behind procurement cycles


Open Source Solutions

• Several software projects on the internet:
– Relax and recover
– mkCDrec
– Mondo Rescue
– CRU
– BACULA
– Your-Favorite-Selfmade-Script
– <something you know but I never heard about>
• In 2006, all of them where lacking something...


Relax and Recover – ReaR

http://rear.sourceforge.net
• GPL Software – Developers in Germany and Belgium
• 100% Bash script – no GUI and no dependencies
• Utilize kernel, modules, binaries of host (kernel ≥ 2.6)
• Support any combination of SW/HW RAID, LVM
• Internal backup on CIFS, NFS ...
• Boot media on CD/DVD, USB key and LAN (PXE)

• Successor of mkCDrec

ReaR – Features

• Focus on disaster recovery and not backup
• Tight integration with common backup software –
delegate file backup to backup infrastructure
• Simple full backup integrated
• Complements backup software:
– Backup software: Data storage and retrieval
– ReaR: Recover system layout and make it work again
– ReaR utilizes the backup software to restore the backup data

• Use the best tool for the job

ReaR – Backup Software

• Supported solutions include:
– CommVault Galaxy
– IBM Tivoli Storage Manager
– Veritas NetBackup
– HP Data Protector
– Rsync and other “external” methods
– tar.gz archive on NAS share – CIFS, NFS, NCP ...
• Very transparent integration
• Quick implementation: 2-3 PT programming
• Can be easily extended to support other vendors

ReaR – Network Integration

• Disaster recovery as part of network infrastructure
– Backup software – file-level backup storage
– ReaR – system environment
– Boot rescue media via PXE – no physical media required
– Very scalable – automated installation of entire disaster recovery
data center

ReaR
Boot Files Software
Backup
PXE Boot
Backup
Network Storage
infrastructure

ReaR – Status

• Stable software
– i386 and x86_64 well tested
– ia64 and ppc experimental
• Regular releases (RPM, DEB, TGZ)
• Major Linux distributions ship ReaR:
– SUSE Linux Enterprise High Availability Extension 11 SP1
®

– openSUSE 11.2 and Fedora 11
®

• Community and commercial support available
• Regular patch submissions from ReaR users


ReaR - Development

• Open Source development model:
– Submit patches and feedback – “field testing”
– Sponsoring
• Modular design:
– ReaR is a framework to plug together many small bash scripts
– Maximize code reusability
– Simple development model (vi works fine)
– Little to no “interferences” between different areas of code
• Documentation on project homepage
– Hierarchical structure
– References to source code


ReaR – Configuration

• Shell-script style configuration file
• Usually: Set backup software and boot media

System definition:
ARCH = Linux-x86_64
OS = GNU/Linux
OS_VENDOR = SUSE_LINUX
OS_VERSION = 11
OS_VENDOR_ARCH = SUSE_LINUX/x86_64
OS_VENDOR_VERSION = SUSE_LINUX/11
OS_VENDOR_VERSION_ARCH = SUSE_LINUX/11/x86_64
Configuration tree:
Linux-x86_64.conf : OK
GNU/Linux.conf : OK
SUSE_LINUX.conf : missing/empty
SUSE_LINUX/x86_64.conf : missing/empty
SUSE_LINUX/11.conf : missing/empty
SUSE_LINUX/11/x86_64.conf : missing/empty
site.conf : OK
local.conf : OK

ReaR – Operation

rear [Options] <command> [command options ...]
Relax & Recover Version 1.7.23.2 / 2010-03-19
Build: b80d32a97e41a6ec8a5b19cd6399e444
Copyright (C) 2006-2009
Schlomo Schapiro
Gratien D'haese, IT3 Consultants
Relax & Recover comes with ABSOLUTELY NO WARRANTY; for details
see the GNU General Public License at http://www.gnu.org/licenses/gpl.html

Available Options:
-V version information
-d debug mode
-D debugscript mode
-S Step-by-step mode
-s Simulation mode (shows the scripts included)
-q Quiet mode
-r a.b.c-xx-yy kernel version to use (current: 2.6.32.8-0.3-default)


ReaR – Operation (continued)

List of commands:
dump Dump configuration and system information
help print out usage
mkbackup Create rescue media and backup system.
mkbackuponly Backup system without creating a (new)
rescue media.
mkdeb Create DEB packages with this rear version
mkdist Create distribution tar archive with this
rear version
mkrescue Create rescue media only
mkrpm Create RPM packages with this rear version
mktar Create tar archive with this rear
installation
mkvendorrpm Create vendor RPM with this rear version
recover Recover the system
validate Submit validation information


ReaR – Example Execution
# rear mkbackup
The preparation phase OK
Physical devices that will be recovered: /dev/sda /dev/sdb
Creating root FS layout OK
Copy files and directories OK
Copy program files & libraries OK
Copy kernel modules OK
Create initramfs OK
Creating archive 'nfs://server.domain/backup-path/rear/backup.tar.gz'
Transferred 297 MB in 254 seconds [1197 KB/sec]
Making ISO image OK
Wrote ISO Image /tmp/ReaR.iso (19M)
Copying resulting files to network location OK
The cleanup phase OK
Finished in 258 seconds.


ReaR – Disk Layout
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
2,5G 927M 1,4G 40% /
devtmpfs 244M 128K 244M 1% /dev
tmpfs 247M 0 247M 0% /dev/shm
/dev/sda1 84M 19M 61M 24% /boot
/dev/mapper/system-home
524M 17M 481M 4% /home
/dev/md0 680M 55M 591M 9% /var


ReaR – Disk Layout

# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 system lvm2 a- 3,21G 0
/dev/sdb2 system lvm2 a- 312,00M 0

# lvs
LV VG Attr LSize Origin Snap%
home system -wi-ao 532,00M
root system -wi-ao 2,44G
swap system -wi-ao 572,00M

# cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sda2[0] sdb1[1]
706816 blocks super 1.0 [2/2] [UU]


ReaR – Recovery Example

# rear recover
Backup archive size is 298M
Initializing physical devices OK
Creating Software RAID devices OK
Creating the LVM2 devices OK
Creating file systems OK
Recreated this filesystem layout:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
2.5G 68M 2.3G 3% /mnt/local
/dev/sda1 84M 5.6M 74M 7% /mnt/local/boot
/dev/mapper/system-home
524M 17M 481M 4% /mnt/local/home
/dev/md0 680M 17M 629M 3% /mnt/local/var
Restoring from 'nfs://laptop/dev/shm/nfs/rear/backup.tar.gz' OK


ReaR – Recovery Example (continued)

Restoring the bootloader (SuSE style)
Creating swap files and partitions OK
Finished recovering your system

You can explore it under /mnt/local

Thank you for using Relax & Recover Version 1.7.23.2

Finished in 150 seconds.


ReaR – Vision

• Do more than just disaster recovery
– System cloning and provisioning
– System migration (P2V, V2P, V2V, P2P)
– Generic system tool-kit for offline tasks and rescue media
• Centralized information and management:
– rear-server
– 1.0: Gather and aggregate information about ReaR
> System protection status
> Rescue media boot files
– 2.0: Centrally manage configuration and control ReaR
> Site-wide and system-local configuration store
> Trigger and control disaster recovery tasks
– 3.0: Integrate into 3rd party solutions


Secure Disaster Recovery
Enterprise Environments

Enterprise DR – Requirements

Standardization
Scalability
Automation
Integration in existing system environment
Security
Coverage over all systems in use
Support and maintenance
Education and certification
Affordable licensing – site license


Enterprise DR – Relax and Recover

• Standardization through global configuration:
– /etc/rear/site.conf
• Scalability and automation:
– rear mkrescue can be run as a cron job
– Automated processing of rescue boot media fairly simple
– Rescue boot media also stored with backup software
– LAN boot – extreme scalability and automated DR possible
• Integration in existing system environment
– Support internal and external backup solutions
– E-mail reports

Enterprise DR – Relax and Recover

• Security considerations
– No “secret” data in rescue media
– Secure export of rescue media through backup software
and E-mail
– Remote access to rescue system only with SSH keys
• Coverage
– i386 and x86_64 stable
– Enterprise Linux (Novell , Red Hat, Ubuntu) stable
®

– Community distributions also work, less tested
• Maintenance and support, education
– Please contact Novell and the authors
• Site license: Included with GPL

Security Aspects – Disaster Recovery

Backup server DR boot server

DR client

Transfer of Backup Data Transfer of Boot Riles
• How secure is the • Insecure (anonymous):
backup software? – NFS
• Insecure: NFS2, NFS3 – CIFS
• Secure (with – SSH (keys)
authentication): – RCP
– NFS4, CIFS • Manipulation proof:
– SSH (rsync) – SMTP

Enterprise Linux Disaster Recovery

• Linux – Simple Disaster Recovery
• Integration: Disaster recovery – backup software
• Automated creation of disaster recovery information
• Network boot into rescue system
• Disaster recovery becomes part of infrastructure
• Scales well even for very large data centers
• Disaster recovery becomes a valid option for clients,
not only for servers


SUSE Linux Enterprise High Availability
®

Extension 11 Service Pack 11 –
ReaR Availability and Support

SUSE Linux Enterprise
®

Modular Portfolio

SUSE Linux Enterprise

Common Code Level 3 Support Runs on x86,
Base foundation x86_64, Itanium,
Power, and s390x

SUSE Linux SLES for SAP,
Enterprise SLED JeOS SLE SDK SLE POS Expanded
Server Support

Server Desktop Custom Developer Bundles Support
OS OS OS Tools SKUs

High Availability Real Time Extension Mono Extension

Optional extensions for high availability clustering, Pay only for
deterministic computing, and running .NET applications what you need


SUSE Linux Enterprise High Availability Extension 11
®

Key Features
• Flexible, policy driven • Continuous data
clustering solution replication
– OpenAIS messaging and – Distributed Replicated Block
membership layer Device (DRBD8)
– Pacemaker cluster
resource manager • GUI and CLI tools
– Graphical user interface
• Cluster-aware file system – Unified command line interface
and volume manager
– Oracle Cluster File • IP load balancing
System (OCFS2)
– Clustered Logical Volume
• Resource agents
Manager (cLVM2)
• Virtualization aware
• Bare Metal Recovery (ReaR)


®

Differentiators
• Affordable
Costs less than other Linux clustering solutions, with tools
and resource agents included at no extra charge

• Virtualization-aware
Supports hybrid virtual and physical clusters

• Open Source
No proprietary modules or user space applications means your
Linux clustering solution is secure and easily kept up to date

• Integrated
Streamlined support and an optimized solution from a single
vendor that delivers both the OS and clustering software


®

New Features/Enhancements SP1
ReaR – Relax and Recover
• ReaR will be included with SUSE Linux Entreprise
Server 11 SP1 HA Add-on
• ReaR is considered to be an extensible framework, allowing
for custom scripting and extension of the recovery toolset
• ReaR RPM packages will be fully supported (L1, L2, L3)
• SLE11 High Availability Add-on inherits service level
agreement from the SLES Subscription
– Priority (24x7)
– Standard (12x5)
• Customization is not covered by Novell Technical Support
®

– Integration Partners are available (contact Novell Sales)
•

®

More Information - SP1
For more information on what is new with SLE11 HA SP1:

ELS307 Building High Availability Clusters with SUSE
Linux Enterprise High Availability Extension

ELS302 SUSE Linux Enterprise High Availability
Extension 11: Support and Troubleshooting

ATT307 Quickly Configure a High Availability Cluster with
SUSE Linux Enterprise 11 High Availability Extension


Relax and Recover – Live Demo

Demo (~10 min):
• SUSE Linux ®

Enterprise High
Availability Extension
11 SP1
• ReaR creates backup
and boot CD
• “PEBKAC” at work
• Boot system from CD
• Recover system
• Happy End

Unpublished Work of Novell, Inc. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc.
Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope
of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified,
translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in
making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents
of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any
particular purpose. The development, release, and timing of features or functionality described for Novell products
remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to
make changes to its content, at any time, without obligation to notify any person or entity of such revisions or
changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc.
in the United States and other countries. All third-party trademarks are the property of their respective owners.

Linux Disaster Recovery Made Easy

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Ähnlich wie Linux Disaster Recovery Made Easy

Ähnlich wie Linux Disaster Recovery Made Easy (20)

Mehr von Novell

Mehr von Novell (20)

Linux Disaster Recovery Made Easy