SlideShare ist ein Scribd-Unternehmen logo
1 von 59
Downloaden Sie, um offline zu lesen
Open Source Data Backup, or:
How To Sleep Better At Night
Fran Fabrizio
Senior Systems Administrator
Dept. of Computer and Information Sciences
University of Alabama at Birmingham
O’Reilly Open Source Convention, August 1-5, 2005
Open Source Data Backup (OSCON 2005), Slide 2
Talk Overview
• Introduction to Amanda
– What it is, how it thinks
• Amanda In Action
– Real world examples
• Configuring Amanda
– Quick look at the config files
• Introduction to Bacula
– What it is, how it thinks
• Bacula vs. Amanda
– Major differences
• For More Info
Open Source Data Backup (OSCON 2005), Slide 3
Introduction To Amanda
What is Amanda?
How does it work?
What is its philosophy?
Open Source Data Backup (OSCON 2005), Slide 4
What is Amanda?
• Cross-Platform
• Scalable
• Automated
• Flexible
• Robust
Amanda is the Advanced Maryland Automatic
Network Disk Archiver. It has the following
features:
Open Source Data Backup (OSCON 2005), Slide 5
What is Amanda? (Cont)
• A set of CLI utilities written in C
• Its own protocols on top of TCP and UDP
• Client and server components that partner to
stream and store data for backup and recovery
Open Source Data Backup (OSCON 2005), Slide 6
Holding
Disk
Tape
Drive
OS X Client
Solaris Client
Linux Client Windows
Client
Amanda
Amanda Server
amandad
Samba
amandad
amandad
Sample Topology
Open Source Data Backup (OSCON 2005), Slide 7
Typical Sequence of Events
Amanda Server Amanda Client
amdump
planner
dumper(s)
driver
taper
amandad
amandad
launches
passes schedule
spawns
writes
to disk
writes to
taper
flush to tape
requests estimate
returns estimate
requests backup
returns image
(many clients serviced in parallel)
Open Source Data Backup (OSCON 2005), Slide 8
How Does It Work?
• gtar and dump
• Uses standard backup levels
• Manages tapes
• Balances resources
• Supports compression and encryption
• Degrades gracefully
Open Source Data Backup (OSCON 2005), Slide 9
Some Amanda Terminology
• Dump cycle
– How often do you want a full backup?
• Disklist Entry / DLE / Target
– Something (partition, filesystem) you want to backup
Open Source Data Backup (OSCON 2005), Slide 10
Amanda's Philosophy
• “You tell me how often you want a full backup,
and I'll worry about everything else”
• At least one full backup of each DLE per cycle
• You don't get to say when full backups happen
• Sounds scary, but usually sufficient
Open Source Data Backup (OSCON 2005), Slide 11
Real World Examples – CIS @ UAB
• Before....
(And this isn't even the whole dirty truth)
Open Source Data Backup (OSCON 2005), Slide 12
Real World Examples – CIS @ UAB
• So we bought...
Great! Only
one problem...
Open Source Data Backup (OSCON 2005), Slide 13
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
• ....in Alabama ;-)
• We had no more money
• Amanda saved the day
• Usage at CIS
– 43 filesystems/partitions on 14 Linux, Solaris and
Windows clients
– ~ 60GB of data per night
– Fully automated, requiring about 2 hours of attention so
far this year
Open Source Data Backup (OSCON 2005), Slide 14
Real World Examples - Others
• Other Examples
– Much Larger
• One user wrote about a 700GB nightly dump
– Much Smaller
• one client systems
– Vtape setups
• virtual tapes on disk
• periodically burn to DVD
Open Source Data Backup (OSCON 2005), Slide 15
Typical Amanda Daily Operation
• At 2:30pm, am check utility runs via cron job and
informs me via email if there are any problems
• At 2:00am, am dum p utility runs to kick off the
Amanda backup process
• Sometime in the morning, amdump sends an email
summarizing last night's activity.
Open Source Data Backup (OSCON 2005), Slide 16
Sample amdump Email Output
These dumps were to tape CIS-004.
The next tape Amanda expects to use is: CIS-005.
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 1:00
Run Time (hrs:min) 2:57
Dump Time (hrs:min) 3:24 2:02 1:22
Output Size (meg) 20957.1 17678.4 3278.8
Original Size (meg) 38473.8 31408.7 7065.1
Avg Compressed Size (%) 54.5 56.3 46.4 (level:#disks ...)
Filesystems Dumped 41 11 30 (1:28 3:1 4:1)
Avg Dump Rate (k/s) 1753.8 2474.3 682.5
Tape Time (hrs:min) 0:21 0:15 0:05
Tape Size (meg) 20957.2 17678.4 3278.8
Tape Used (%) 4.3 3.6 0.7 (level:#disks ...)
Filesystems Taped 41 11 30 (1:28 3:1 4:1)
Avg Tp Write Rate (k/s) 17361.9 19931.6 10242.2
USAGE BY TAPE:
Label Time Size % Nb
CIS-004 0:21 20957.2 4.3 41
Open Source Data Backup (OSCON 2005), Slide 17
Sample amdump Email Output
(Continued)NOTES:
planner: Full dump of virginia:/hc promoted from 4 days ahead.
planner: Full dump of florida:/var promoted from 4 days ahead.
planner: Full dump of alabama:/etc promoted from 4 days ahead.
planner: Full dump of florida:/etc promoted from 1 day ahead.
planner: Full dump of alabama:/home promoted from 1 day ahead.
planner: Full dump of georgia:/home promoted from 4 days ahead.
planner: Full dump of illinois:/home promoted from 4 days ahead.
planner: Full dump of newyork:/home promoted from 4 days ahead.
planner: Full dump of newjersey:/ promoted from 4 days ahead.
taper: tape CIS-004 kb 21462176 fm 41 [OK]
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- --------------------------------- ------------
missouri.hpcl.c -xport/home 1 24671 13205 53.5 3:42 59.6 0:018848.1
missouri.hpcl.c /export/opt 1 1055 49 4.6 0:44 1.1 0:01 42.8
missouri.hpcl.c /var/mail 1 1538303 828724 53.9 13:171040.4 1:299345.3
nevada.cis.ua / 4 2271760 881675 38.8 5:092850.4 1:0813033.1
ohio.cis.u /home 1 1540 247 16.0 0:01 401.8 0:04 69.7
maine.cis. /home 1 82640 59078 71.5 0:144300.2 0:087671.2
florida.cis.u /etc 0 5200 1613 31.0 0:03 625.5 0:011186.9
florida.cis.u /he 1 8780 614 7.0 0:26 23.4 0:01 512.8
florida.cis.u /hf 1 16440 2066 12.6 2:02 16.9 0:021249.4
etc....
(brought to you by Amanda version 2.4.4p2)
Open Source Data Backup (OSCON 2005), Slide 18
Amanda's planner In Action
planner clientA
“Please estimate level 0, level 1 and level 2 backups for /home.”
Let's assume that
last night Amanda
performed a level 1
backup of clientA's
/home DLE...
clientA returns the info
“Will promoting this to a level 0 (full) dump lead to
more balance over the dump cycle?”
“Will we save a significant amount of tape space by
going to a level 2 incremental instead of level 1?”
Schedules
Level 0 dump
Schedules
Level 2 dump
Schedules level 1 dump
Yes
Yes
No
No
Open Source Data Backup (OSCON 2005), Slide 19
More about planner
• Guarantees one full dump per cycle
• If the tape is too small for the run's data, it will
delay some of the dumps in the least disruptive
way
• Looks at past dumps to determine optimal
balancing behavior
• Tries to stay as close to level 0 as possible to
reduce need to use multiple tapes for restore
Open Source Data Backup (OSCON 2005), Slide 20
Restoring One Or A Few Files
[root@alabam a /tm p]# amrecover -C CIS -s
amanda.cis.uab.edu -t amanda.cis.uab.edu -d
/dev/nst0
AMRECOVER Version 2.4.4p1.Contacting server on
am anda.cis.uab.edu ...
220 keep AMANDA index server (2.4.4p2) ready.
200 Access OK
Setting restore date to today (2004-06-28)
200 Working date set to 2004-06-28.
Scanning /dum ps/am anda...
20040622:found Am anda directory.
200 Config set to CIS.
200 Dum p host set to alabam a.cis.uab.edu.
am recover> setdisk /var/spool/mail
200 Disk set to /var/spool/m ail.
am recover> setdate 2004-06-25
200 Working date set to 2004-06-25.
Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25.
Solution: Use the amrecover utility
Open Source Data Backup (OSCON 2005), Slide 21
Restoring One Or A Few Files (Con't)
am recover> ls
[...]
2004-06-25 brockhw
2004-06-25 brownta
2004-06-25 bryant
2004-06-25 byrdv
[...]
am recover> add bryant
Added /bryant
am recover> extract
Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.
The following tapes are needed:CIS-024
Restoring files into directory /tm p
Continue [?/Y/n]? Y
Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.
Load tape CIS-024 now
Continue [?/Y/n/s/t]? Y
./bryant
am recover> quit
200 Good bye.
[root@alabam a /tm p]# ls -l bryant
-rw------- 1 bryant disk 14533946 Jun 24 19:10 bryant
Open Source Data Backup (OSCON 2005), Slide 22
Restoring An Entire Backup Target
Problem: The disk holding /etc on a critical server has failed
Solution: Use the amrestore utility.
[root@am anda testrestore]# amadmin CIS info alabama.cis.uab.edu '/etc$'
Current info for alabam a.cis.uab.edu /etc:
Stats:dum p rates (kps),Full: 481.0,320.0,350.0
Increm ental: 19.0, 13.0, 11.0
com pressed size,Full: 20.6%,20.6%,20.6%
Increm ental: 5.4%, 5.4%, 5.4%
Dum ps:lev datestm p tape file origK com pK secs
0 20040623 CIS-022 23 18690 3849 8
1 20040628 CIS-027 10 700 38 0
[root@am anda testrestore]# su - amanda
-bash-2.05b$ am tape CIS labelCIS-022
am tape:scanning for tape with labelCIS-022
am tape:slot 26:date 20040623 labelCIS-022 (exact labelm atch)
am tape:labelCIS-022 is now loaded.
-bash-2.05b$ exit
Open Source Data Backup (OSCON 2005), Slide 23
Restoring An Entire Backup Target (Con't)
[root@am anda testrestore]# amrestore -p /dev/nst0 alabama.cis.uab.edu /etc >
etc.0.tar
am restore: 0:skipping start oftape:date 20040623 labelCIS-022
am restore: 1:skipping am anda.cis.uab.edu.__texas_dfs_hom e_undergrad.private.20040623.1
am restore: 2:skipping florida.cis.uab.edu._etc.20040623.1
am restore: 3:skipping alabam a.cis.uab.edu._root.20040623.1
[...]
amrestore: 21: skipping missouri.cis.uab.edu._.20040623.0
amrestore: 22: skipping oregon.cis.uab.edu._home.20040623.1
amrestore: 23: restoring alabama.cis.uab.edu._etc.20040623.0
amrestore: 24: reached end of information
[root@am anda testrestore]# ls -l
total18716
-rw-r--r-- 1 root root 19138560 Jun 28 14:40 etc.0.tar
Open Source Data Backup (OSCON 2005), Slide 24
Other Ways To Restore Files
• Amanda not available?
root@am anda testrestore]# mt -f /dev/nst0 rewind
[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1
AMANDA:TAPESTART DATE 20040623 TAPE CIS-022
1+0 records in
1+0 records out
[root@am anda testrestore]# mt -f /dev/nst0 fsf 01
[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1
AMANDA:FILE 20040623 am anda.cis.uab.edu
//texas/dfs/hom e/undergrad.public lev 1 com p .gz program /usr/bin/sm bclient
To restore,position tape at start offile and run:
dd if=<tape> bs=32k skip=1 |/bin/gzip -dc |usr/bin/sm bclient -f...-
1+0 records in
1+0 records out
Open Source Data Backup (OSCON 2005), Slide 25
When Things Go Wrong
• Data backup is a complex interaction between a
lot of players, and things -will- go wrong...
– Hosts will be down, or away (laptops)
– Tapes will go bad
– You'll change a password and then forget to tell Amanda
you changed it
– Your holding disk might be too small
• Amanda tries to be proactive by running am check
and giving you time to fix problems
Open Source Data Backup (OSCON 2005), Slide 26
Sample amcheck Problem Report
• Host down
Amanda Tape Server Host Check
-----------------------------
Holding disk /dumps/amanda: 60967688 KB disk space available, that's plenty
amcheck-server: slot 5: date 20040520 label CIS-018 (exact label match)
NOTE: skipping tape-writable test
Tape CIS-018 label ok
Server check took 175.627 seconds
Amanda Backup Client Hosts Check
--------------------------------
WARNING: vermont.cis.uab.edu: selfcheck request timed out. Host down?
Client check: 13 hosts checked in 30.210 seconds, 1 problem found
(brought to you by Amanda 2.4.4p2)
Open Source Data Backup (OSCON 2005), Slide 27
More amcheck Sample Messages
NOTE: index dir
/
usr/local/etc/amanda/CIS/index/amanda.cis.uab.edu/__texas_dfs_home_undergr
ad.private: does not exist
ERROR: georgia.cis.uab.edu: [access as amanda not allowed from
amanda@amanda] amandahostsauth failed
ERROR: virginia.cis.uab.edu: [dir /etc needs 64KB, only has 5KB
WARNING: holding disk /dumps/amanda: only 50254708 KB free (52428800 KB
requested)
amcheck-server: slot 19: rewinding tape: No medium found
amcheck-server: slot 19: date 20040330 label CIS-027 (active tape)
amcheck-server: fatal slot 20: slot 20 move failed
ERROR: label CIS-016 or new tape not found in rack
(expecting tape CIS-016 or a new tape)
amcheck-server: could not get changer info: could not read result from
"/usr/local/libexec/chg-scsi"
WARNING: skipping tape test because amdump or amflush seem to be running
WARNING: if they are not, you must run amcleanup
Open Source Data Backup (OSCON 2005), Slide 28
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
• Run this script manually any time you touch a
config file
• Run this script manually any time you add or
change a client
• It will save you hours of troubleshooting
• One of the best features of Amanda
Open Source Data Backup (OSCON 2005), Slide 29
amcheck Can't Do Everything
• am check cannot catch problems that arise after it
runs
• am check doesn't check for everything
• Amanda's other utilities are really good at telling
you why they could not do their job
Open Source Data Backup (OSCON 2005), Slide 30
Revisiting The amdump Email Report
• Very rarely is the email report as uneventful as
was presented earlier. It typically will include
something like this:
These dumps were to tape CIS-004.
The next tape Amanda expects to use is: CIS-005.
FAILURE AND STRANGE DUMP SUMMARY:
alabama.cis.u /usr lev 1 STRANGE
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 1:00
Run Time (hrs:min) 2:57
etc....
Open Source Data Backup (OSCON 2005), Slide 31
Revisiting The amdump Email Report (Con't)
• And then later on, it will explain itself....
FAILED AND STRANGE DUMP DETAILS:
/-- alabama.cis.u /usr lev 1 STRANGE
sendbackup: start [alabama.cis.uab.edu:/usr level 1]
sendbackup: info BACKUP=/bin/gtar
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/gtar -f... -
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
? gtar: ./local/majordomo-1.94.5/log/mdlog: file changed as we
read it
| Total bytes written: 207923200 (198MB, 3.1MB/s)
sendbackup: size 203050
sendbackup: end
--------
Amanda is telling us a file was in use as it tried to
grab it. This is usually harmless and constitutes
99% of STRANGE results.
Open Source Data Backup (OSCON 2005), Slide 32
Revisiting The amdump Email Report
(Con't)
• Sometimes, there are more serious failures
FAILURE AND STRANGE DUMP SUMMARY:
amanda.cis.u //texas/dfs/simnetxpcd lev 0 FAILED [disk
//texas/dfs/simnetxpcd, all estimate failed]
amanda.cis.u //texas/dfs/classfiles lev 0 FAILED [disk
//texas/dfs/classfiles, all estimate failed]
amanda.cis.u //texas/dfs/officefiles lev 0 FAILED [disk
//texas/dfs/officefiles, all estimate failed]
amanda.cis.u //texas/dfs/scripts lev 0 FAILED [disk //texas/dfs/scripts,
all estimate failed]
amanda.cis.u //texas/dfs/home/graduate.private lev 0 FAILED [disk
//texas/dfs/home/graduate.private, all estimate failed]
amanda.cis.u //texas/dfs/home/graduate.public lev 0 FAILED [disk
//texas/dfs/home/graduate.public, all estimate failed]
amanda.cis.u //texas/dfs/home/undergrad.private lev 0 FAILED [disk
//texas/dfs/home/undergrad.private, all estimate failed]
This was because I changed the password to the SAMBA
share that Amanda was using to back up a Windows server.
The email clued me in and the problem was resolved
quickly.
Open Source Data Backup (OSCON 2005), Slide 33
Handling Tape Failures
• Amanda keeps going and stores as much as
possible in the holding disk
• You can then use am flush to flush data to tape
-bash-2.05b$ amflush CIS
Scanning /dum ps/am anda...
 20040516:found Am anda directory.
 20040517:found Am anda directory.
Multiple Am anda directories,please pick one by letter:
 A.20040516
 B.20040517
Select directories to flush [A..B]:[ALL]
Today is:20040517
Flushing dum ps in 20040516,20040517 using tape changer "chg-scsi".
Expecting tape CIS-015 or a new tape. (The last dum ps were to tape CIS-014)
Are you sure you want to do this [yN]? y
Running in background,you can log offnow.
You'llget m ailwhen am flush is finished.
Open Source Data Backup (OSCON 2005), Slide 34
Amanda Prerequisites
• A server that is mostly idle during the times that
you want to do your backups
• Enough disk space for a suitable holding disk
• GNU tar
• Samba (for Windows clients)
• A large capacity tape drive (typically)
• GNUplot
Open Source Data Backup (OSCON 2005), Slide 35
Configuration Roadmap
• Set up server
– Create amanda user and assign to group with permission
to use the tape and changer devices (e.g. 'disk' on Linux)
– Gather info on tape and changer devices - mt and mtx
are handy scripts to have here
– Open ports (10080,10082,10083) and set up services
(amanda, amandaidx, amidxtape)
– Configure amanda.conf & changer (e.g. chg-scsi.conf)
• Set up clients
– Create amanda user
– Config ports, services, files, directories
– Allow access (typically .amandahosts)
Open Source Data Backup (OSCON 2005), Slide 36
Things To Think About Before
Proceeding
• What should my cycle be?
– Once per night? Every three nights?
• If your cycle seems convoluted, try using two:
– One for daily backups, once per night, one week cycle
– One for archives, always full dumps, run manually when
you need it
• Find the right balance for the cycle length
– Short cycles eat up lots of resources doing full dumps
– Long cycles can be a pain to restore from, you might
need 4 tapes for example
• How many tapes to use?
– How far back into the past do you want to go?
Open Source Data Backup (OSCON 2005), Slide 37
amanda.conf
• Amanda's main configuration file
• Many options
• Well documented
• Cannot begin to cover everything here, definitely
read the documentation first!
• This is where you define your cycle's parameters
(length, number of runs, number of tapes, etc...)
Open Source Data Backup (OSCON 2005), Slide 38
amanda.conf - dumptypes
• Different rules for each DLE
define dum ptype root-tar {
global
program "GNUTAR"
com m ent "root partitions dum ped with tar"
com press none
index yes
exclude list "/usr/local/lib/am anda/exclude.gtar"
priority low
}
define dum ptype com p-high {
global
com m ent "very im portant partitions on fast m achines"
com press client best
priority high
}
Open Source Data Backup (OSCON 2005), Slide 39
disklist
• Defining your DLEs / targets
# File form at is:
#
# hostnam e diskdev dum ptype [spindle [interface]]
# the tape server itself
am anda.cis.uab.edu / com p-root-tar-exclude-holdingdisk
# the directory server
newjersey.cis.uab.edu / com p-root-tar
# the file server
virginia.cis.uab.edu /m z com p-user-tar
virginia.cis.uab.edu /root com p-root-tar
virginia.cis.uab.edu /usr com p-user-tar
virginia.cis.uab.edu /hc com p-root-tar
Open Source Data Backup (OSCON 2005), Slide 40
Configuring Tape Changers
• Amanda has a generic interface to tape changers
• Tape changer configuration is stored in chg-
scsi.conf
• chg-scsi.conf is one of many changer scripts that
come with Amanda
• This is where you tell Amanda how many drives
you have, which tapes go to which drives, barcode
support, etc...
Open Source Data Backup (OSCON 2005), Slide 41
Tape Drive Configuration Heads-Up
• There are many choices of changer configuration
scripts (chg-scsi, chg-multi, chg-mtx, chg-manual,
chg-disk, etc...)
• You may have to use one even if you don't have
an actual changer (chg-manual, chg-disk)
• Many options, many chances for confusion
• Please read docs/TAPE.CHANGERS and other
sources of information (listed at end of talk)
Open Source Data Backup (OSCON 2005), Slide 42
Configuring Your Tape Collection
• Use amlabel to label new tapes
– am labelCIS CIS-000 slot 0
• Then use amtape to build the tapelist
– am tape CIS update
am tape:scanning all30 slots in tape-changer rack:
slot 26:date 20040623 labelCIS-022
etc...
Open Source Data Backup (OSCON 2005), Slide 43
Special Case Configurations
• Windows Clients
– Use Samba to backup Windows clients
– Configure the Amanda server or other Unix client with
Samba shares
– Then in the disklist point to that server and share
•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
Open Source Data Backup (OSCON 2005), Slide 44
Special Case Configurations
• Windows Clients
– Use Samba to backup Windows clients
– Configure the Amanda server or other Unix client with
Samba shares
– Then in the disklist point to that server and share
•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
• Firewalls
– --with-tcpportrange=40000,40030 ( something > 1024)
– --with-udpportrange=920,940 (something < 1024)
– or, iptables has amanda support you can enable
Open Source Data Backup (OSCON 2005), Slide 45
Introduction To Bacula
What is Bacula?
How is it different?
Open Source Data Backup (OSCON 2005), Slide 46
What is Bacula?
• “The Network Backup Tool for Linux, Unix, Mac
and Windows.”
• Another open-source project that aims to provide
a robust network-based, multiplatform backup
solution
• Newer than Amanda (started in 2000)
“It comes in the night and sucks the essence from your computers.”
- Kern Sibbald
Open Source Data Backup (OSCON 2005), Slide 47
Bacula Features
• Modular, scalable components
• Its own protocols on top of TCP and UDP
• Client and server components that partner to
stream and store data for backup and recovery
• Clean component separation - all communication
between them goes over the network
• Threaded rather than multiple processes
• Excellent documentation
Open Source Data Backup (OSCON 2005), Slide 48
How is Bacula Different than Amanda?
• It can support multiple volumes
• You may find it easier to setup
• There are both command line and GUI
configuration tools available
• Scheduler gives you more control over what jobs
run at which times
• It will reuse a tape on multiple nights until it is
full
• Support for automated restores from bare metal
• Native support for Windows (no Samba/NFS)
• SQL database support
Open Source Data Backup (OSCON 2005), Slide 49
Sample Bacula Topology
Director
Daemon
Database
Server
File Daemon
Windows
File Daemon
Linux
File Daemon
Unix
File Daemon
OS X
Storage
Daemon
Tape
Device
Storage
Daemon
Tape
Changer
Storage
Daemon
Disk
Device
Admin
Console
Open Source Data Backup (OSCON 2005), Slide 50
Component Roles
• The Director manages all scheduling and job
creation. It is via an administration interface
talking to the Director that the backup
administrator controls the backup process.
• The Storage Daemon is responsible for writing
data out to disk/tape/changer
• The Database keeps the catalog of what has been
backed up and where
• The File Daemon streams the data to be backed up
from the client to the Storage Daemon
Open Source Data Backup (OSCON 2005), Slide 51
Prerequisites
• Bacula currently works with SQLite, MySQL and
PostgreSQL
• GNU C++ 2.95 or higher to compile
• Other software may be necessary depending on
configuration. If burning DVDs, you need the
dvd+rw-tools. If using the GUI console, you need
recent GNOME and GTK+ libs.
Open Source Data Backup (OSCON 2005), Slide 52
Configuration Overview
• Bacula is configured via a series of config files
– bacula-dir.conf, bacula-fd.conf, bacula-sd.conf,
console.conf
• Examples coming here
Open Source Data Backup (OSCON 2005), Slide 53
Storage Organization
• Bacula organizes tapes (volumes) into pools. It
will use one volume up until it is full, and then
move on to the next. You can give more control if
you want to use a new tape each night
• Similar to Amanda, each Bacula volume get a
unique label and added to the pool. However,
unlike Amanda, there is no set rotation. Bacula
will use one until full, and then go look for
another, etc...
• Can use multiple pools to ensure that a new tape is
used each day. e.q. Setup a Monday pool, a
Tuesday pool, etc...
Open Source Data Backup (OSCON 2005), Slide 54
Pre and Post Job Scripts
• Bacula's File Daemon has the ability to run a
script before and after a job
• This can be used to shutdown a database in order
to take a safe backup of it, for example
• Use the bacula-fd.conf directives “Run Before
Job” and “Run After Job”
Open Source Data Backup (OSCON 2005), Slide 55
Take Home Points...
• Amanda is very robust, highly scalable, almost
infinitely configurable, and can very likely handle
your data backup situation
• If you find Amanda limiting or do not agree with
the scheduling philosophy, Bacula may be for
you. Momentum -may- be headed to Bacula.
• Once you get either of these configured, you can
trust it, and move on with your life. They work,
are battle-tested and just as reliable, if not more
so, than the expensive commercial products.
• Test your backup and recovery system and
strategy early and often!!
Open Source Data Backup (OSCON 2005), Slide 56
For More Information
• http://www.amanda.org/
• Documentation
– README, INSTALL, docs/* (esp. FAQ and
TAPE.CHANGERS), example/*
– The AMANDA section in O'Reilly's “Unix Backup and
Recovery”
– man pages
• http://www.bacula.org/
– Documentation - Tutorial, Quick Start, User Guide
• The user communities are wonderful
– Sign up for amanda-users@amanda.org at
http://www.amanda.org/support/mailinglists.php
– Sign up for bacula-users@lists.sourceforge.net at
http://lists.sourceforge.net/lists/listinfo/bacula-users
Open Source Data Backup (OSCON 2005), Slide 57
My Information
• My email is fran@cis.uab.edu
• This presentation is available at
http://www.cis.uab.edu/fran/
• More Amanda and Bacula information can be
found at the above URL, including:
– A document detailing every step of my Amanda
configuration, along with complete sample amanda.conf
and chg-scsi.conf
– Notes on configuring Samba to backup Windows shares
– Sample Bacula configuration files for each component
– Notes on strategies for setting up volumes and pools for
Bacula
Open Source Data Backup (OSCON 2005), Slide 58
My Information
• My email is fran@cis.uab.edu
• This presentation is available at
http://www.cis.uab.edu/fran/
• Much more Amanda information can be found at
the above URL, including:
– A document detailing every step of my configuration
– A complete sample amanda.conf and chg-scsi.conf
– Notes on configuring Samba to backup Windows shares
The End - Thank You!
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)

Weitere ähnliche Inhalte

Was ist angesagt?

XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...The Linux Foundation
 
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.The Linux Foundation
 
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld
 
MySQL Server Settings Tuning
MySQL Server Settings TuningMySQL Server Settings Tuning
MySQL Server Settings Tuningguest5ca94b
 
PostgreSQL Portland Performance Practice Project - Database Test 2 Howto
PostgreSQL Portland Performance Practice Project - Database Test 2 HowtoPostgreSQL Portland Performance Practice Project - Database Test 2 Howto
PostgreSQL Portland Performance Practice Project - Database Test 2 HowtoMark Wong
 
JavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame GraphsJavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame GraphsBrendan Gregg
 
제3회난공불락 오픈소스 인프라세미나 - lustre
제3회난공불락 오픈소스 인프라세미나 - lustre제3회난공불락 오픈소스 인프라세미나 - lustre
제3회난공불락 오픈소스 인프라세미나 - lustreTommy Lee
 
pg_prefaulter: Scaling WAL Performance
pg_prefaulter: Scaling WAL Performancepg_prefaulter: Scaling WAL Performance
pg_prefaulter: Scaling WAL PerformanceSean Chittenden
 
6048618 cloning-procedure-of-r12-single-tier
6048618 cloning-procedure-of-r12-single-tier6048618 cloning-procedure-of-r12-single-tier
6048618 cloning-procedure-of-r12-single-tierbalaji29
 
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, Arm
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, ArmXPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, Arm
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, ArmThe Linux Foundation
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
 
OWF: Xen - Open Source Hypervisor Designed for Clouds
OWF: Xen - Open Source Hypervisor Designed for CloudsOWF: Xen - Open Source Hypervisor Designed for Clouds
OWF: Xen - Open Source Hypervisor Designed for CloudsThe Linux Foundation
 
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...The Linux Foundation
 
[CCP Games] Versioning Everything with Perforce
[CCP Games] Versioning Everything with Perforce[CCP Games] Versioning Everything with Perforce
[CCP Games] Versioning Everything with PerforcePerforce
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornSimon Bagreev
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
Systems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudSystems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudBrendan Gregg
 

Was ist angesagt? (20)

XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
 
Universal Userland
Universal UserlandUniversal Userland
Universal Userland
 
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
 
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
 
MySQL Server Settings Tuning
MySQL Server Settings TuningMySQL Server Settings Tuning
MySQL Server Settings Tuning
 
PostgreSQL Portland Performance Practice Project - Database Test 2 Howto
PostgreSQL Portland Performance Practice Project - Database Test 2 HowtoPostgreSQL Portland Performance Practice Project - Database Test 2 Howto
PostgreSQL Portland Performance Practice Project - Database Test 2 Howto
 
JavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame GraphsJavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame Graphs
 
제3회난공불락 오픈소스 인프라세미나 - lustre
제3회난공불락 오픈소스 인프라세미나 - lustre제3회난공불락 오픈소스 인프라세미나 - lustre
제3회난공불락 오픈소스 인프라세미나 - lustre
 
pg_prefaulter: Scaling WAL Performance
pg_prefaulter: Scaling WAL Performancepg_prefaulter: Scaling WAL Performance
pg_prefaulter: Scaling WAL Performance
 
6048618 cloning-procedure-of-r12-single-tier
6048618 cloning-procedure-of-r12-single-tier6048618 cloning-procedure-of-r12-single-tier
6048618 cloning-procedure-of-r12-single-tier
 
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, Arm
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, ArmXPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, Arm
XPDDS18: The Art of Virtualizing Cache Maintenance - Julien Grall, Arm
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 
OWF: Xen - Open Source Hypervisor Designed for Clouds
OWF: Xen - Open Source Hypervisor Designed for CloudsOWF: Xen - Open Source Hypervisor Designed for Clouds
OWF: Xen - Open Source Hypervisor Designed for Clouds
 
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...
 
[CCP Games] Versioning Everything with Perforce
[CCP Games] Versioning Everything with Perforce[CCP Games] Versioning Everything with Perforce
[CCP Games] Versioning Everything with Perforce
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and Unicorn
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
LFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and BeyondLFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and Beyond
 
Systems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudSystems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the Cloud
 

Ähnlich wie Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)

Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)
Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)
Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)Fran Fabrizio
 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013Server Density
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganTobias Koprowski
 
KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018Tobias Koprowski
 
What the Heck Just Happened?
What the Heck Just Happened?What the Heck Just Happened?
What the Heck Just Happened?Ken Evans
 
BGOUG "Agile Data: revolutionizing database cloning'
BGOUG  "Agile Data: revolutionizing database cloning'BGOUG  "Agile Data: revolutionizing database cloning'
BGOUG "Agile Data: revolutionizing database cloning'Kyle Hailey
 
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganKoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganTobias Koprowski
 
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganKoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganTobias Koprowski
 
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedEasy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedChenkai Sun
 
Technical track-afterimaging Progress Database
Technical track-afterimaging Progress DatabaseTechnical track-afterimaging Progress Database
Technical track-afterimaging Progress DatabaseVinh Nguyen
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]Kyle Hailey
 
OOUG - Oracle Performance Tuning with AAS
OOUG - Oracle Performance Tuning with AASOOUG - Oracle Performance Tuning with AAS
OOUG - Oracle Performance Tuning with AASKyle Hailey
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationKyle Hailey
 
Death of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopDeath of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopErik Riedel
 
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...cookie1969
 
Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Wing Venture Capital
 
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14Kyle Hailey
 
KoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganKoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganTobias Koprowski
 

Ähnlich wie Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005) (20)

Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)
Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)
Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)
 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
 
KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018
 
What the Heck Just Happened?
What the Heck Just Happened?What the Heck Just Happened?
What the Heck Just Happened?
 
BGOUG "Agile Data: revolutionizing database cloning'
BGOUG  "Agile Data: revolutionizing database cloning'BGOUG  "Agile Data: revolutionizing database cloning'
BGOUG "Agile Data: revolutionizing database cloning'
 
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganKoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
 
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganKoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
 
RMAN – The Pocket Knife of a DBA
RMAN – The Pocket Knife of a DBA RMAN – The Pocket Knife of a DBA
RMAN – The Pocket Knife of a DBA
 
Hptf 2240 Final
Hptf 2240 FinalHptf 2240 Final
Hptf 2240 Final
 
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedEasy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
 
Technical track-afterimaging Progress Database
Technical track-afterimaging Progress DatabaseTechnical track-afterimaging Progress Database
Technical track-afterimaging Progress Database
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
 
OOUG - Oracle Performance Tuning with AAS
OOUG - Oracle Performance Tuning with AASOOUG - Oracle Performance Tuning with AAS
OOUG - Oracle Performance Tuning with AAS
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualization
 
Death of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopDeath of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO Workshop
 
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
 
Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007
 
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
 
KoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganKoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just began
 

Kürzlich hochgeladen

Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 

Kürzlich hochgeladen (20)

Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 

Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)

  • 1. Open Source Data Backup, or: How To Sleep Better At Night Fran Fabrizio Senior Systems Administrator Dept. of Computer and Information Sciences University of Alabama at Birmingham O’Reilly Open Source Convention, August 1-5, 2005
  • 2. Open Source Data Backup (OSCON 2005), Slide 2 Talk Overview • Introduction to Amanda – What it is, how it thinks • Amanda In Action – Real world examples • Configuring Amanda – Quick look at the config files • Introduction to Bacula – What it is, how it thinks • Bacula vs. Amanda – Major differences • For More Info
  • 3. Open Source Data Backup (OSCON 2005), Slide 3 Introduction To Amanda What is Amanda? How does it work? What is its philosophy?
  • 4. Open Source Data Backup (OSCON 2005), Slide 4 What is Amanda? • Cross-Platform • Scalable • Automated • Flexible • Robust Amanda is the Advanced Maryland Automatic Network Disk Archiver. It has the following features:
  • 5. Open Source Data Backup (OSCON 2005), Slide 5 What is Amanda? (Cont) • A set of CLI utilities written in C • Its own protocols on top of TCP and UDP • Client and server components that partner to stream and store data for backup and recovery
  • 6. Open Source Data Backup (OSCON 2005), Slide 6 Holding Disk Tape Drive OS X Client Solaris Client Linux Client Windows Client Amanda Amanda Server amandad Samba amandad amandad Sample Topology
  • 7. Open Source Data Backup (OSCON 2005), Slide 7 Typical Sequence of Events Amanda Server Amanda Client amdump planner dumper(s) driver taper amandad amandad launches passes schedule spawns writes to disk writes to taper flush to tape requests estimate returns estimate requests backup returns image (many clients serviced in parallel)
  • 8. Open Source Data Backup (OSCON 2005), Slide 8 How Does It Work? • gtar and dump • Uses standard backup levels • Manages tapes • Balances resources • Supports compression and encryption • Degrades gracefully
  • 9. Open Source Data Backup (OSCON 2005), Slide 9 Some Amanda Terminology • Dump cycle – How often do you want a full backup? • Disklist Entry / DLE / Target – Something (partition, filesystem) you want to backup
  • 10. Open Source Data Backup (OSCON 2005), Slide 10 Amanda's Philosophy • “You tell me how often you want a full backup, and I'll worry about everything else” • At least one full backup of each DLE per cycle • You don't get to say when full backups happen • Sounds scary, but usually sufficient
  • 11. Open Source Data Backup (OSCON 2005), Slide 11 Real World Examples – CIS @ UAB • Before.... (And this isn't even the whole dirty truth)
  • 12. Open Source Data Backup (OSCON 2005), Slide 12 Real World Examples – CIS @ UAB • So we bought... Great! Only one problem...
  • 13. Open Source Data Backup (OSCON 2005), Slide 13 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university... • ....in Alabama ;-) • We had no more money • Amanda saved the day • Usage at CIS – 43 filesystems/partitions on 14 Linux, Solaris and Windows clients – ~ 60GB of data per night – Fully automated, requiring about 2 hours of attention so far this year
  • 14. Open Source Data Backup (OSCON 2005), Slide 14 Real World Examples - Others • Other Examples – Much Larger • One user wrote about a 700GB nightly dump – Much Smaller • one client systems – Vtape setups • virtual tapes on disk • periodically burn to DVD
  • 15. Open Source Data Backup (OSCON 2005), Slide 15 Typical Amanda Daily Operation • At 2:30pm, am check utility runs via cron job and informs me via email if there are any problems • At 2:00am, am dum p utility runs to kick off the Amanda backup process • Sometime in the morning, amdump sends an email summarizing last night's activity.
  • 16. Open Source Data Backup (OSCON 2005), Slide 16 Sample amdump Email Output These dumps were to tape CIS-004. The next tape Amanda expects to use is: CIS-005. STATISTICS: Total Full Daily -------- -------- -------- Estimate Time (hrs:min) 1:00 Run Time (hrs:min) 2:57 Dump Time (hrs:min) 3:24 2:02 1:22 Output Size (meg) 20957.1 17678.4 3278.8 Original Size (meg) 38473.8 31408.7 7065.1 Avg Compressed Size (%) 54.5 56.3 46.4 (level:#disks ...) Filesystems Dumped 41 11 30 (1:28 3:1 4:1) Avg Dump Rate (k/s) 1753.8 2474.3 682.5 Tape Time (hrs:min) 0:21 0:15 0:05 Tape Size (meg) 20957.2 17678.4 3278.8 Tape Used (%) 4.3 3.6 0.7 (level:#disks ...) Filesystems Taped 41 11 30 (1:28 3:1 4:1) Avg Tp Write Rate (k/s) 17361.9 19931.6 10242.2 USAGE BY TAPE: Label Time Size % Nb CIS-004 0:21 20957.2 4.3 41
  • 17. Open Source Data Backup (OSCON 2005), Slide 17 Sample amdump Email Output (Continued)NOTES: planner: Full dump of virginia:/hc promoted from 4 days ahead. planner: Full dump of florida:/var promoted from 4 days ahead. planner: Full dump of alabama:/etc promoted from 4 days ahead. planner: Full dump of florida:/etc promoted from 1 day ahead. planner: Full dump of alabama:/home promoted from 1 day ahead. planner: Full dump of georgia:/home promoted from 4 days ahead. planner: Full dump of illinois:/home promoted from 4 days ahead. planner: Full dump of newyork:/home promoted from 4 days ahead. planner: Full dump of newjersey:/ promoted from 4 days ahead. taper: tape CIS-004 kb 21462176 fm 41 [OK] DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -------------------------- --------------------------------- ------------ missouri.hpcl.c -xport/home 1 24671 13205 53.5 3:42 59.6 0:018848.1 missouri.hpcl.c /export/opt 1 1055 49 4.6 0:44 1.1 0:01 42.8 missouri.hpcl.c /var/mail 1 1538303 828724 53.9 13:171040.4 1:299345.3 nevada.cis.ua / 4 2271760 881675 38.8 5:092850.4 1:0813033.1 ohio.cis.u /home 1 1540 247 16.0 0:01 401.8 0:04 69.7 maine.cis. /home 1 82640 59078 71.5 0:144300.2 0:087671.2 florida.cis.u /etc 0 5200 1613 31.0 0:03 625.5 0:011186.9 florida.cis.u /he 1 8780 614 7.0 0:26 23.4 0:01 512.8 florida.cis.u /hf 1 16440 2066 12.6 2:02 16.9 0:021249.4 etc.... (brought to you by Amanda version 2.4.4p2)
  • 18. Open Source Data Backup (OSCON 2005), Slide 18 Amanda's planner In Action planner clientA “Please estimate level 0, level 1 and level 2 backups for /home.” Let's assume that last night Amanda performed a level 1 backup of clientA's /home DLE... clientA returns the info “Will promoting this to a level 0 (full) dump lead to more balance over the dump cycle?” “Will we save a significant amount of tape space by going to a level 2 incremental instead of level 1?” Schedules Level 0 dump Schedules Level 2 dump Schedules level 1 dump Yes Yes No No
  • 19. Open Source Data Backup (OSCON 2005), Slide 19 More about planner • Guarantees one full dump per cycle • If the tape is too small for the run's data, it will delay some of the dumps in the least disruptive way • Looks at past dumps to determine optimal balancing behavior • Tries to stay as close to level 0 as possible to reduce need to use multiple tapes for restore
  • 20. Open Source Data Backup (OSCON 2005), Slide 20 Restoring One Or A Few Files [root@alabam a /tm p]# amrecover -C CIS -s amanda.cis.uab.edu -t amanda.cis.uab.edu -d /dev/nst0 AMRECOVER Version 2.4.4p1.Contacting server on am anda.cis.uab.edu ... 220 keep AMANDA index server (2.4.4p2) ready. 200 Access OK Setting restore date to today (2004-06-28) 200 Working date set to 2004-06-28. Scanning /dum ps/am anda... 20040622:found Am anda directory. 200 Config set to CIS. 200 Dum p host set to alabam a.cis.uab.edu. am recover> setdisk /var/spool/mail 200 Disk set to /var/spool/m ail. am recover> setdate 2004-06-25 200 Working date set to 2004-06-25. Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25. Solution: Use the amrecover utility
  • 21. Open Source Data Backup (OSCON 2005), Slide 21 Restoring One Or A Few Files (Con't) am recover> ls [...] 2004-06-25 brockhw 2004-06-25 brownta 2004-06-25 bryant 2004-06-25 byrdv [...] am recover> add bryant Added /bryant am recover> extract Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu. The following tapes are needed:CIS-024 Restoring files into directory /tm p Continue [?/Y/n]? Y Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu. Load tape CIS-024 now Continue [?/Y/n/s/t]? Y ./bryant am recover> quit 200 Good bye. [root@alabam a /tm p]# ls -l bryant -rw------- 1 bryant disk 14533946 Jun 24 19:10 bryant
  • 22. Open Source Data Backup (OSCON 2005), Slide 22 Restoring An Entire Backup Target Problem: The disk holding /etc on a critical server has failed Solution: Use the amrestore utility. [root@am anda testrestore]# amadmin CIS info alabama.cis.uab.edu '/etc$' Current info for alabam a.cis.uab.edu /etc: Stats:dum p rates (kps),Full: 481.0,320.0,350.0 Increm ental: 19.0, 13.0, 11.0 com pressed size,Full: 20.6%,20.6%,20.6% Increm ental: 5.4%, 5.4%, 5.4% Dum ps:lev datestm p tape file origK com pK secs 0 20040623 CIS-022 23 18690 3849 8 1 20040628 CIS-027 10 700 38 0 [root@am anda testrestore]# su - amanda -bash-2.05b$ am tape CIS labelCIS-022 am tape:scanning for tape with labelCIS-022 am tape:slot 26:date 20040623 labelCIS-022 (exact labelm atch) am tape:labelCIS-022 is now loaded. -bash-2.05b$ exit
  • 23. Open Source Data Backup (OSCON 2005), Slide 23 Restoring An Entire Backup Target (Con't) [root@am anda testrestore]# amrestore -p /dev/nst0 alabama.cis.uab.edu /etc > etc.0.tar am restore: 0:skipping start oftape:date 20040623 labelCIS-022 am restore: 1:skipping am anda.cis.uab.edu.__texas_dfs_hom e_undergrad.private.20040623.1 am restore: 2:skipping florida.cis.uab.edu._etc.20040623.1 am restore: 3:skipping alabam a.cis.uab.edu._root.20040623.1 [...] amrestore: 21: skipping missouri.cis.uab.edu._.20040623.0 amrestore: 22: skipping oregon.cis.uab.edu._home.20040623.1 amrestore: 23: restoring alabama.cis.uab.edu._etc.20040623.0 amrestore: 24: reached end of information [root@am anda testrestore]# ls -l total18716 -rw-r--r-- 1 root root 19138560 Jun 28 14:40 etc.0.tar
  • 24. Open Source Data Backup (OSCON 2005), Slide 24 Other Ways To Restore Files • Amanda not available? root@am anda testrestore]# mt -f /dev/nst0 rewind [root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1 AMANDA:TAPESTART DATE 20040623 TAPE CIS-022 1+0 records in 1+0 records out [root@am anda testrestore]# mt -f /dev/nst0 fsf 01 [root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1 AMANDA:FILE 20040623 am anda.cis.uab.edu //texas/dfs/hom e/undergrad.public lev 1 com p .gz program /usr/bin/sm bclient To restore,position tape at start offile and run: dd if=<tape> bs=32k skip=1 |/bin/gzip -dc |usr/bin/sm bclient -f...- 1+0 records in 1+0 records out
  • 25. Open Source Data Backup (OSCON 2005), Slide 25 When Things Go Wrong • Data backup is a complex interaction between a lot of players, and things -will- go wrong... – Hosts will be down, or away (laptops) – Tapes will go bad – You'll change a password and then forget to tell Amanda you changed it – Your holding disk might be too small • Amanda tries to be proactive by running am check and giving you time to fix problems
  • 26. Open Source Data Backup (OSCON 2005), Slide 26 Sample amcheck Problem Report • Host down Amanda Tape Server Host Check ----------------------------- Holding disk /dumps/amanda: 60967688 KB disk space available, that's plenty amcheck-server: slot 5: date 20040520 label CIS-018 (exact label match) NOTE: skipping tape-writable test Tape CIS-018 label ok Server check took 175.627 seconds Amanda Backup Client Hosts Check -------------------------------- WARNING: vermont.cis.uab.edu: selfcheck request timed out. Host down? Client check: 13 hosts checked in 30.210 seconds, 1 problem found (brought to you by Amanda 2.4.4p2)
  • 27. Open Source Data Backup (OSCON 2005), Slide 27 More amcheck Sample Messages NOTE: index dir / usr/local/etc/amanda/CIS/index/amanda.cis.uab.edu/__texas_dfs_home_undergr ad.private: does not exist ERROR: georgia.cis.uab.edu: [access as amanda not allowed from amanda@amanda] amandahostsauth failed ERROR: virginia.cis.uab.edu: [dir /etc needs 64KB, only has 5KB WARNING: holding disk /dumps/amanda: only 50254708 KB free (52428800 KB requested) amcheck-server: slot 19: rewinding tape: No medium found amcheck-server: slot 19: date 20040330 label CIS-027 (active tape) amcheck-server: fatal slot 20: slot 20 move failed ERROR: label CIS-016 or new tape not found in rack (expecting tape CIS-016 or a new tape) amcheck-server: could not get changer info: could not read result from "/usr/local/libexec/chg-scsi" WARNING: skipping tape test because amdump or amflush seem to be running WARNING: if they are not, you must run amcleanup
  • 28. Open Source Data Backup (OSCON 2005), Slide 28 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office • Run this script manually any time you touch a config file • Run this script manually any time you add or change a client • It will save you hours of troubleshooting • One of the best features of Amanda
  • 29. Open Source Data Backup (OSCON 2005), Slide 29 amcheck Can't Do Everything • am check cannot catch problems that arise after it runs • am check doesn't check for everything • Amanda's other utilities are really good at telling you why they could not do their job
  • 30. Open Source Data Backup (OSCON 2005), Slide 30 Revisiting The amdump Email Report • Very rarely is the email report as uneventful as was presented earlier. It typically will include something like this: These dumps were to tape CIS-004. The next tape Amanda expects to use is: CIS-005. FAILURE AND STRANGE DUMP SUMMARY: alabama.cis.u /usr lev 1 STRANGE STATISTICS: Total Full Daily -------- -------- -------- Estimate Time (hrs:min) 1:00 Run Time (hrs:min) 2:57 etc....
  • 31. Open Source Data Backup (OSCON 2005), Slide 31 Revisiting The amdump Email Report (Con't) • And then later on, it will explain itself.... FAILED AND STRANGE DUMP DETAILS: /-- alabama.cis.u /usr lev 1 STRANGE sendbackup: start [alabama.cis.uab.edu:/usr level 1] sendbackup: info BACKUP=/bin/gtar sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/gtar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: ./local/majordomo-1.94.5/log/mdlog: file changed as we read it | Total bytes written: 207923200 (198MB, 3.1MB/s) sendbackup: size 203050 sendbackup: end -------- Amanda is telling us a file was in use as it tried to grab it. This is usually harmless and constitutes 99% of STRANGE results.
  • 32. Open Source Data Backup (OSCON 2005), Slide 32 Revisiting The amdump Email Report (Con't) • Sometimes, there are more serious failures FAILURE AND STRANGE DUMP SUMMARY: amanda.cis.u //texas/dfs/simnetxpcd lev 0 FAILED [disk //texas/dfs/simnetxpcd, all estimate failed] amanda.cis.u //texas/dfs/classfiles lev 0 FAILED [disk //texas/dfs/classfiles, all estimate failed] amanda.cis.u //texas/dfs/officefiles lev 0 FAILED [disk //texas/dfs/officefiles, all estimate failed] amanda.cis.u //texas/dfs/scripts lev 0 FAILED [disk //texas/dfs/scripts, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.private lev 0 FAILED [disk //texas/dfs/home/graduate.private, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.public lev 0 FAILED [disk //texas/dfs/home/graduate.public, all estimate failed] amanda.cis.u //texas/dfs/home/undergrad.private lev 0 FAILED [disk //texas/dfs/home/undergrad.private, all estimate failed] This was because I changed the password to the SAMBA share that Amanda was using to back up a Windows server. The email clued me in and the problem was resolved quickly.
  • 33. Open Source Data Backup (OSCON 2005), Slide 33 Handling Tape Failures • Amanda keeps going and stores as much as possible in the holding disk • You can then use am flush to flush data to tape -bash-2.05b$ amflush CIS Scanning /dum ps/am anda...  20040516:found Am anda directory.  20040517:found Am anda directory. Multiple Am anda directories,please pick one by letter:  A.20040516  B.20040517 Select directories to flush [A..B]:[ALL] Today is:20040517 Flushing dum ps in 20040516,20040517 using tape changer "chg-scsi". Expecting tape CIS-015 or a new tape. (The last dum ps were to tape CIS-014) Are you sure you want to do this [yN]? y Running in background,you can log offnow. You'llget m ailwhen am flush is finished.
  • 34. Open Source Data Backup (OSCON 2005), Slide 34 Amanda Prerequisites • A server that is mostly idle during the times that you want to do your backups • Enough disk space for a suitable holding disk • GNU tar • Samba (for Windows clients) • A large capacity tape drive (typically) • GNUplot
  • 35. Open Source Data Backup (OSCON 2005), Slide 35 Configuration Roadmap • Set up server – Create amanda user and assign to group with permission to use the tape and changer devices (e.g. 'disk' on Linux) – Gather info on tape and changer devices - mt and mtx are handy scripts to have here – Open ports (10080,10082,10083) and set up services (amanda, amandaidx, amidxtape) – Configure amanda.conf & changer (e.g. chg-scsi.conf) • Set up clients – Create amanda user – Config ports, services, files, directories – Allow access (typically .amandahosts)
  • 36. Open Source Data Backup (OSCON 2005), Slide 36 Things To Think About Before Proceeding • What should my cycle be? – Once per night? Every three nights? • If your cycle seems convoluted, try using two: – One for daily backups, once per night, one week cycle – One for archives, always full dumps, run manually when you need it • Find the right balance for the cycle length – Short cycles eat up lots of resources doing full dumps – Long cycles can be a pain to restore from, you might need 4 tapes for example • How many tapes to use? – How far back into the past do you want to go?
  • 37. Open Source Data Backup (OSCON 2005), Slide 37 amanda.conf • Amanda's main configuration file • Many options • Well documented • Cannot begin to cover everything here, definitely read the documentation first! • This is where you define your cycle's parameters (length, number of runs, number of tapes, etc...)
  • 38. Open Source Data Backup (OSCON 2005), Slide 38 amanda.conf - dumptypes • Different rules for each DLE define dum ptype root-tar { global program "GNUTAR" com m ent "root partitions dum ped with tar" com press none index yes exclude list "/usr/local/lib/am anda/exclude.gtar" priority low } define dum ptype com p-high { global com m ent "very im portant partitions on fast m achines" com press client best priority high }
  • 39. Open Source Data Backup (OSCON 2005), Slide 39 disklist • Defining your DLEs / targets # File form at is: # # hostnam e diskdev dum ptype [spindle [interface]] # the tape server itself am anda.cis.uab.edu / com p-root-tar-exclude-holdingdisk # the directory server newjersey.cis.uab.edu / com p-root-tar # the file server virginia.cis.uab.edu /m z com p-user-tar virginia.cis.uab.edu /root com p-root-tar virginia.cis.uab.edu /usr com p-user-tar virginia.cis.uab.edu /hc com p-root-tar
  • 40. Open Source Data Backup (OSCON 2005), Slide 40 Configuring Tape Changers • Amanda has a generic interface to tape changers • Tape changer configuration is stored in chg- scsi.conf • chg-scsi.conf is one of many changer scripts that come with Amanda • This is where you tell Amanda how many drives you have, which tapes go to which drives, barcode support, etc...
  • 41. Open Source Data Backup (OSCON 2005), Slide 41 Tape Drive Configuration Heads-Up • There are many choices of changer configuration scripts (chg-scsi, chg-multi, chg-mtx, chg-manual, chg-disk, etc...) • You may have to use one even if you don't have an actual changer (chg-manual, chg-disk) • Many options, many chances for confusion • Please read docs/TAPE.CHANGERS and other sources of information (listed at end of talk)
  • 42. Open Source Data Backup (OSCON 2005), Slide 42 Configuring Your Tape Collection • Use amlabel to label new tapes – am labelCIS CIS-000 slot 0 • Then use amtape to build the tapelist – am tape CIS update am tape:scanning all30 slots in tape-changer rack: slot 26:date 20040623 labelCIS-022 etc...
  • 43. Open Source Data Backup (OSCON 2005), Slide 43 Special Case Configurations • Windows Clients – Use Samba to backup Windows clients – Configure the Amanda server or other Unix client with Samba shares – Then in the disklist point to that server and share •am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
  • 44. Open Source Data Backup (OSCON 2005), Slide 44 Special Case Configurations • Windows Clients – Use Samba to backup Windows clients – Configure the Amanda server or other Unix client with Samba shares – Then in the disklist point to that server and share •am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar • Firewalls – --with-tcpportrange=40000,40030 ( something > 1024) – --with-udpportrange=920,940 (something < 1024) – or, iptables has amanda support you can enable
  • 45. Open Source Data Backup (OSCON 2005), Slide 45 Introduction To Bacula What is Bacula? How is it different?
  • 46. Open Source Data Backup (OSCON 2005), Slide 46 What is Bacula? • “The Network Backup Tool for Linux, Unix, Mac and Windows.” • Another open-source project that aims to provide a robust network-based, multiplatform backup solution • Newer than Amanda (started in 2000) “It comes in the night and sucks the essence from your computers.” - Kern Sibbald
  • 47. Open Source Data Backup (OSCON 2005), Slide 47 Bacula Features • Modular, scalable components • Its own protocols on top of TCP and UDP • Client and server components that partner to stream and store data for backup and recovery • Clean component separation - all communication between them goes over the network • Threaded rather than multiple processes • Excellent documentation
  • 48. Open Source Data Backup (OSCON 2005), Slide 48 How is Bacula Different than Amanda? • It can support multiple volumes • You may find it easier to setup • There are both command line and GUI configuration tools available • Scheduler gives you more control over what jobs run at which times • It will reuse a tape on multiple nights until it is full • Support for automated restores from bare metal • Native support for Windows (no Samba/NFS) • SQL database support
  • 49. Open Source Data Backup (OSCON 2005), Slide 49 Sample Bacula Topology Director Daemon Database Server File Daemon Windows File Daemon Linux File Daemon Unix File Daemon OS X Storage Daemon Tape Device Storage Daemon Tape Changer Storage Daemon Disk Device Admin Console
  • 50. Open Source Data Backup (OSCON 2005), Slide 50 Component Roles • The Director manages all scheduling and job creation. It is via an administration interface talking to the Director that the backup administrator controls the backup process. • The Storage Daemon is responsible for writing data out to disk/tape/changer • The Database keeps the catalog of what has been backed up and where • The File Daemon streams the data to be backed up from the client to the Storage Daemon
  • 51. Open Source Data Backup (OSCON 2005), Slide 51 Prerequisites • Bacula currently works with SQLite, MySQL and PostgreSQL • GNU C++ 2.95 or higher to compile • Other software may be necessary depending on configuration. If burning DVDs, you need the dvd+rw-tools. If using the GUI console, you need recent GNOME and GTK+ libs.
  • 52. Open Source Data Backup (OSCON 2005), Slide 52 Configuration Overview • Bacula is configured via a series of config files – bacula-dir.conf, bacula-fd.conf, bacula-sd.conf, console.conf • Examples coming here
  • 53. Open Source Data Backup (OSCON 2005), Slide 53 Storage Organization • Bacula organizes tapes (volumes) into pools. It will use one volume up until it is full, and then move on to the next. You can give more control if you want to use a new tape each night • Similar to Amanda, each Bacula volume get a unique label and added to the pool. However, unlike Amanda, there is no set rotation. Bacula will use one until full, and then go look for another, etc... • Can use multiple pools to ensure that a new tape is used each day. e.q. Setup a Monday pool, a Tuesday pool, etc...
  • 54. Open Source Data Backup (OSCON 2005), Slide 54 Pre and Post Job Scripts • Bacula's File Daemon has the ability to run a script before and after a job • This can be used to shutdown a database in order to take a safe backup of it, for example • Use the bacula-fd.conf directives “Run Before Job” and “Run After Job”
  • 55. Open Source Data Backup (OSCON 2005), Slide 55 Take Home Points... • Amanda is very robust, highly scalable, almost infinitely configurable, and can very likely handle your data backup situation • If you find Amanda limiting or do not agree with the scheduling philosophy, Bacula may be for you. Momentum -may- be headed to Bacula. • Once you get either of these configured, you can trust it, and move on with your life. They work, are battle-tested and just as reliable, if not more so, than the expensive commercial products. • Test your backup and recovery system and strategy early and often!!
  • 56. Open Source Data Backup (OSCON 2005), Slide 56 For More Information • http://www.amanda.org/ • Documentation – README, INSTALL, docs/* (esp. FAQ and TAPE.CHANGERS), example/* – The AMANDA section in O'Reilly's “Unix Backup and Recovery” – man pages • http://www.bacula.org/ – Documentation - Tutorial, Quick Start, User Guide • The user communities are wonderful – Sign up for amanda-users@amanda.org at http://www.amanda.org/support/mailinglists.php – Sign up for bacula-users@lists.sourceforge.net at http://lists.sourceforge.net/lists/listinfo/bacula-users
  • 57. Open Source Data Backup (OSCON 2005), Slide 57 My Information • My email is fran@cis.uab.edu • This presentation is available at http://www.cis.uab.edu/fran/ • More Amanda and Bacula information can be found at the above URL, including: – A document detailing every step of my Amanda configuration, along with complete sample amanda.conf and chg-scsi.conf – Notes on configuring Samba to backup Windows shares – Sample Bacula configuration files for each component – Notes on strategies for setting up volumes and pools for Bacula
  • 58. Open Source Data Backup (OSCON 2005), Slide 58 My Information • My email is fran@cis.uab.edu • This presentation is available at http://www.cis.uab.edu/fran/ • Much more Amanda information can be found at the above URL, including: – A document detailing every step of my configuration – A complete sample amanda.conf and chg-scsi.conf – Notes on configuring Samba to backup Windows shares The End - Thank You!