SlideShare ist ein Scribd-Unternehmen logo
1 von 89
Downloaden Sie, um offline zu lesen
Amanda for Data Backup, or:
How To Sleep Better At Night
Fran Fabrizio
Senior Systems Administrator
Dept. of Computer and Information Sciences
University of Alabama at Birmingham
fran@cis.uab.edu
http://www.cis.uab.edu/fran/
O’Reilly Open Source Convention, July 26–30, 2004
Amanda for Data Backup (OSCON 2004), Slide 2
Talk Overview
• Introduction to Amanda
– What it is, how it thinks
• Amanda In Action
– Real world examples
• Configuring Amanda
– Quick look at the config files
• Other Features
• Take Home Lessons
• For More Info
Amanda for Data Backup (OSCON 2004), Slide 3
Introduction To Amanda
What is Amanda?
How does it work?
What is its philosophy?
Amanda for Data Backup (OSCON 2004), Slide 4
What is Amanda?
• Cross-Platform
• Scalable
• Automated
• Flexible
• Robust
Amanda is the Advanced Maryland Automatic
Network Disk Archiver. It has the following
features:
Amanda for Data Backup (OSCON 2004), Slide 5
What is Amanda? (Cont)
• A set of CLI utilities written in C
• Its own protocols on top of TCP and UDP
• Client and server components that partner to
stream and store data for backup and recovery
Amanda for Data Backup (OSCON 2004), Slide 6
Holding
Disk
Tape
Drive
OS X Client
Solaris Client
Linux Client Windows
Client
Amanda
Amanda Server
amandad
Samba
amandad
amandad
Sample Topology
Amanda for Data Backup (OSCON 2004), Slide 7
Typical Sequence of Events
Amanda Server Amanda Client
amdump
planner
dumper(s)
driver
taper
amandad
amandad
launches
passes schedule
spawns
writes
to disk
writes to
taper
flush to tape
requests estimate
returns estimate
requests backup
returns image
(many clients serviced in parallel)
Amanda for Data Backup (OSCON 2004), Slide 8
How Does It Work?
• gtar and dump
• Uses standard backup levels
• Manages tapes
• Balances resources
• Supports compression and encryption
• Degrades gracefully
Amanda for Data Backup (OSCON 2004), Slide 9
Some Amanda Terminology
• Dump cycle
– How often do you want a full backup?
• Disklist Entry / DLE / Target
– Something (partition, filesystem) you want to backup
Amanda for Data Backup (OSCON 2004), Slide 10
Amanda's Philosophy
• “You tell me how often you want a full backup,
and I'll worry about everything else”
• At least one full backup of each DLE per cycle
• You don't get to say when full backups happen
• Sounds scary, but usually sufficient
– (When it's not, you can hack around it)
Amanda for Data Backup (OSCON 2004), Slide 11
Amanda In Action
What are some real-world examples?
What does typical usage look like?
How do I recover a file or partition?
What happens when things go wrong?
Amanda for Data Backup (OSCON 2004), Slide 12
Real World Examples – CIS @ UAB
• Before....
Amanda for Data Backup (OSCON 2004), Slide 13
Real World Examples – CIS @ UAB
• Before....
Amanda for Data Backup (OSCON 2004), Slide 14
Real World Examples – CIS @ UAB
• Before....
Amanda for Data Backup (OSCON 2004), Slide 15
Real World Examples – CIS @ UAB
• Before....
Amanda for Data Backup (OSCON 2004), Slide 16
Real World Examples – CIS @ UAB
• Before....
Amanda for Data Backup (OSCON 2004), Slide 17
Real World Examples – CIS @ UAB
• Before....
(And this isn't even the whole dirty truth)
Amanda for Data Backup (OSCON 2004), Slide 18
Real World Examples – CIS @ UAB
• Before....
I WORRIED CONSTANTLY AND GOT NO SLEE
Amanda for Data Backup (OSCON 2004), Slide 19
Real World Examples – CIS @ UAB
• Before....
I WORRIED CONSTANTLY AND GOT NO SLEE
(I was also exhibiting
bizarre behavior...)
Amanda for Data Backup (OSCON 2004), Slide 20
Real World Examples – CIS @ UAB
• So we bought...
Amanda for Data Backup (OSCON 2004), Slide 21
Real World Examples – CIS @ UAB
• So we bought...
Great! Only
one problem...
Amanda for Data Backup (OSCON 2004), Slide 22
Real World Examples – CIS @ UAB
• This hardware is very expensive
Amanda for Data Backup (OSCON 2004), Slide 23
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
Amanda for Data Backup (OSCON 2004), Slide 24
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
• ....in Alabama ;-)
Amanda for Data Backup (OSCON 2004), Slide 25
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
• ....in Alabama ;-)
• We had no more money
Amanda for Data Backup (OSCON 2004), Slide 26
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
• ....in Alabama ;-)
• We had no more money
• Amanda saved the day
Amanda for Data Backup (OSCON 2004), Slide 27
Real World Examples – CIS @ UAB
• This hardware is very expensive
• I work for a public university...
• ....in Alabama ;-)
• We had no more money
• Amanda saved the day
• Usage at CIS
– 43 filesystems/partitions on 14 Linux, Solaris and
Windows clients
– ~ 60GB of data per night
– Fully automated, requiring about 30 minutes of attention
in the first half of 2004
Amanda for Data Backup (OSCON 2004), Slide 28
Real World Examples - Others
• Other Examples
– Much Larger
• One user wrote about a 700GB nightly dump
– Much Smaller
• one client systems
– Vtape setups
• virtual tapes on disk
• periodically burn to DVD
Amanda for Data Backup (OSCON 2004), Slide 29
Typical Amanda Daily Operation
• At 2:30pm, am check utility runs via cron job and
informs me via email if there are any problems
Amanda for Data Backup (OSCON 2004), Slide 30
Typical Amanda Daily Operation
• At 2:30pm, am check utility runs via cron job and
informs me via email if there are any problems
• At 2:00am, am dum p utility runs to kick off the
Amanda backup process
Amanda for Data Backup (OSCON 2004), Slide 31
Typical Amanda Daily Operation
• At 2:30pm, am check utility runs via cron job and
informs me via email if there are any problems
• At 2:00am, am dum p utility runs to kick off the
Amanda backup process
• Sometime in the morning, an email arrives
summarizing last night's activity.
Amanda for Data Backup (OSCON 2004), Slide 32
Sample amdump Email Output
These dumps were to tape CIS-004.
The next tape Amanda expects to use is: CIS-005.
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 1:00
Run Time (hrs:min) 2:57
Dump Time (hrs:min) 3:24 2:02 1:22
Output Size (meg) 20957.1 17678.4 3278.8
Original Size (meg) 38473.8 31408.7 7065.1
Avg Compressed Size (%) 54.5 56.3 46.4 (level:#disks ...)
Filesystems Dumped 41 11 30 (1:28 3:1 4:1)
Avg Dump Rate (k/s) 1753.8 2474.3 682.5
Tape Time (hrs:min) 0:21 0:15 0:05
Tape Size (meg) 20957.2 17678.4 3278.8
Tape Used (%) 4.3 3.6 0.7 (level:#disks ...)
Filesystems Taped 41 11 30 (1:28 3:1 4:1)
Avg Tp Write Rate (k/s) 17361.9 19931.6 10242.2
USAGE BY TAPE:
Label Time Size % Nb
CIS-004 0:21 20957.2 4.3 41
Amanda for Data Backup (OSCON 2004), Slide 33
Sample amdump Email Output
(Continued)NOTES:
planner: Full dump of virginia:/hc promoted from 4 days ahead.
planner: Full dump of florida:/var promoted from 4 days ahead.
planner: Full dump of alabama:/etc promoted from 4 days ahead.
planner: Full dump of florida:/etc promoted from 1 day ahead.
planner: Full dump of alabama:/home promoted from 1 day ahead.
planner: Full dump of georgia:/home promoted from 4 days ahead.
planner: Full dump of illinois:/home promoted from 4 days ahead.
planner: Full dump of newyork:/home promoted from 4 days ahead.
planner: Full dump of newjersey:/ promoted from 4 days ahead.
taper: tape CIS-004 kb 21462176 fm 41 [OK]
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- --------------------------------- ------------
missouri.hpcl.c -xport/home 1 24671 13205 53.5 3:42 59.6 0:018848.1
missouri.hpcl.c /export/opt 1 1055 49 4.6 0:44 1.1 0:01 42.8
missouri.hpcl.c /var/mail 1 1538303 828724 53.9 13:171040.4 1:299345.3
nevada.cis.ua / 4 2271760 881675 38.8 5:092850.4 1:0813033.1
ohio.cis.u /home 1 1540 247 16.0 0:01 401.8 0:04 69.7
maine.cis. /home 1 82640 59078 71.5 0:144300.2 0:087671.2
florida.cis.u /etc 0 5200 1613 31.0 0:03 625.5 0:011186.9
florida.cis.u /he 1 8780 614 7.0 0:26 23.4 0:01 512.8
florida.cis.u /hf 1 16440 2066 12.6 2:02 16.9 0:021249.4
etc....
(brought to you by Amanda version 2.4.4p2)
Amanda for Data Backup (OSCON 2004), Slide 34
Amanda's planner In Action
planner clientA
“Please estimate level 0, level 1 and level 2 backups for /home.”
Let's assume that
last night Amanda
performed a level 1
backup of clientA's
/home DLE...
clientA returns the info
“Will promoting this to a level 0 (full) dump lead to
more balance over the dump cycle?”
“Will we save a significant amount of tape space by
going to a level 2 incremental instead of level 1?”
Schedules
Level 0 dump
Schedules
Level 2 dump
Schedules level 1 dump
Yes
Yes
No
No
Amanda for Data Backup (OSCON 2004), Slide 35
More about planner
• Guarantees one full dump per cycle
Amanda for Data Backup (OSCON 2004), Slide 36
More about planner
• Guarantees one full dump per cycle
• If the tape is too small for the run's data, it will
delay some of the dumps in the least disruptive
way
Amanda for Data Backup (OSCON 2004), Slide 37
More about planner
• Guarantees one full dump per cycle
• If the tape is too small for the run's data, it will
delay some of the dumps in the least disruptive
way
• Looks at past dumps to determine optimal
balancing behavior
Amanda for Data Backup (OSCON 2004), Slide 38
More about planner
• Guarantees one full dump per cycle
• If the tape is too small for the run's data, it will
delay some of the dumps in the least disruptive
way
• Looks at past dumps to determine optimal
balancing behavior
• Tries to stay as close to level 0 as possible to
reduce need to use multiple tapes for restore
Amanda for Data Backup (OSCON 2004), Slide 39
Restoring One Or A Few Files
Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25.
Solution: Use the amrecover utility
Amanda for Data Backup (OSCON 2004), Slide 40
Restoring One Or A Few Files
[root@alabam a /tm p]# amrecover -C CIS -s
amanda.cis.uab.edu -t amanda.cis.uab.edu -d
/dev/nst0
AMRECOVER Version 2.4.4p1.Contacting server on
am anda.cis.uab.edu ...
220 keep AMANDA index server (2.4.4p2) ready.
200 Access OK
Setting restore date to today (2004-06-28)
200 Working date set to 2004-06-28.
Scanning /dum ps/am anda...
20040622:found Am anda directory.
200 Config set to CIS.
200 Dum p host set to alabam a.cis.uab.edu.
am recover> setdisk /var/spool/mail
200 Disk set to /var/spool/m ail.
am recover> setdate 2004-06-25
200 Working date set to 2004-06-25.
Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25.
Solution: Use the amrecover utility
Amanda for Data Backup (OSCON 2004), Slide 41
Restoring One Or A Few Files (Con't)
am recover> ls
[...]
2004-06-25 brockhw
2004-06-25 brownta
2004-06-25 bryant
2004-06-25 byrdv
[...]
am recover> add bryant
Added /bryant
am recover> extract
Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.
The following tapes are needed:CIS-024
Restoring files into directory /tm p
Continue [?/Y/n]? Y
Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.
Load tape CIS-024 now
Continue [?/Y/n/s/t]? Y
./bryant
am recover> quit
200 Good bye.
[root@alabam a /tm p]# ls -l bryant
-rw------- 1 bryant disk 14533946 Jun 24 19:10 bryant
Amanda for Data Backup (OSCON 2004), Slide 42
Restoring An Entire Backup Target
Problem: The disk holding /etc on a critical server has failed
Solution: Use the amrestore utility.
Amanda for Data Backup (OSCON 2004), Slide 43
Restoring An Entire Backup Target
Problem: The disk holding /etc on a critical server has failed
Solution: Use the amrestore utility.
[root@am anda testrestore]# amadmin CIS info alabama.cis.uab.edu '/etc$'
Current info for alabam a.cis.uab.edu /etc:
Stats:dum p rates (kps),Full: 481.0,320.0,350.0
Increm ental: 19.0, 13.0, 11.0
com pressed size,Full: 20.6%,20.6%,20.6%
Increm ental: 5.4%, 5.4%, 5.4%
Dum ps:lev datestm p tape file origK com pK secs
0 20040623 CIS-022 23 18690 3849 8
1 20040628 CIS-027 10 700 38 0
[root@am anda testrestore]# su - amanda
-bash-2.05b$ am tape CIS labelCIS-022
am tape:scanning for tape with labelCIS-022
am tape:slot 26:date 20040623 labelCIS-022 (exact labelm atch)
am tape:labelCIS-022 is now loaded.
-bash-2.05b$ exit
Amanda for Data Backup (OSCON 2004), Slide 44
Restoring An Entire Backup Target (Con't)
[root@am anda testrestore]# amrestore -p /dev/nst0 alabama.cis.uab.edu /etc >
etc.0.tar
am restore: 0:skipping start oftape:date 20040623 labelCIS-022
am restore: 1:skipping am anda.cis.uab.edu.__texas_dfs_hom e_undergrad.private.20040623.1
am restore: 2:skipping florida.cis.uab.edu._etc.20040623.1
am restore: 3:skipping alabam a.cis.uab.edu._root.20040623.1
[...]
amrestore: 21: skipping missouri.cis.uab.edu._.20040623.0
amrestore: 22: skipping oregon.cis.uab.edu._home.20040623.1
amrestore: 23: restoring alabama.cis.uab.edu._etc.20040623.0
amrestore: 24: reached end of information
[root@am anda testrestore]# ls -l
total18716
-rw-r--r-- 1 root root 19138560 Jun 28 14:40 etc.0.tar
Amanda for Data Backup (OSCON 2004), Slide 45
Other Ways To Restore Files
• Amanda not available?
Amanda for Data Backup (OSCON 2004), Slide 46
Other Ways To Restore Files
• Amanda not available?
root@am anda testrestore]# mt -f /dev/nst0 rewind
[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1
AMANDA:TAPESTART DATE 20040623 TAPE CIS-022
1+0 records in
1+0 records out
[root@am anda testrestore]# mt -f /dev/nst0 fsf 01
[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1
AMANDA:FILE 20040623 am anda.cis.uab.edu
//texas/dfs/hom e/undergrad.public lev 1 com p .gz program /usr/bin/sm bclient
To restore,position tape at start offile and run:
dd if=<tape> bs=32k skip=1 |/bin/gzip -dc |usr/bin/sm bclient -f...-
1+0 records in
1+0 records out
Amanda for Data Backup (OSCON 2004), Slide 47
When Things Go Wrong
• Data backup is a complex interaction between a
lot of players, and things -will- go wrong...
– Hosts will be down, or away (laptops)
– Tapes will go bad
– You'll change a password and then forget to tell Amanda
you changed it
– Your holding disk might be too small
Amanda for Data Backup (OSCON 2004), Slide 48
When Things Go Wrong
• Data backup is a complex interaction between a
lot of players, and things -will- go wrong...
– Hosts will be down, or away (laptops)
– Tapes will go bad
– You'll change a password and then forget to tell Amanda
you changed it
– Your holding disk might be too small
• Amanda tries to be proactive by running am check
and giving you time to fix problems
Amanda for Data Backup (OSCON 2004), Slide 49
Sample amcheck Problem Report
• Host down
Amanda Tape Server Host Check
-----------------------------
Holding disk /dumps/amanda: 60967688 KB disk space available, that's plenty
amcheck-server: slot 5: date 20040520 label CIS-018 (exact label match)
NOTE: skipping tape-writable test
Tape CIS-018 label ok
Server check took 175.627 seconds
Amanda Backup Client Hosts Check
--------------------------------
WARNING: vermont.cis.uab.edu: selfcheck request timed out. Host down?
Client check: 13 hosts checked in 30.210 seconds, 1 problem found
(brought to you by Amanda 2.4.4p2)
Amanda for Data Backup (OSCON 2004), Slide 50
More amcheck Sample Messages
NOTE: index dir
/
usr/local/etc/amanda/CIS/index/amanda.cis.uab.edu/__texas_dfs_home_undergr
ad.private: does not exist
ERROR: georgia.cis.uab.edu: [access as amanda not allowed from
amanda@amanda] amandahostsauth failed
ERROR: virginia.cis.uab.edu: [dir /etc needs 64KB, only has 5KB
WARNING: holding disk /dumps/amanda: only 50254708 KB free (52428800 KB
requested)
amcheck-server: slot 19: rewinding tape: No medium found
amcheck-server: slot 19: date 20040330 label CIS-027 (active tape)
amcheck-server: fatal slot 20: slot 20 move failed
ERROR: label CIS-016 or new tape not found in rack
(expecting tape CIS-016 or a new tape)
amcheck-server: could not get changer info: could not read result from
"/usr/local/libexec/chg-scsi"
WARNING: skipping tape test because amdump or amflush seem to be running
WARNING: if they are not, you must run amcleanup
Amanda for Data Backup (OSCON 2004), Slide 51
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
Amanda for Data Backup (OSCON 2004), Slide 52
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
• Run this script manually any time you touch a
config file
Amanda for Data Backup (OSCON 2004), Slide 53
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
• Run this script manually any time you touch a
config file
• Run this script manually any time you add or
change a client
Amanda for Data Backup (OSCON 2004), Slide 54
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
• Run this script manually any time you touch a
config file
• Run this script manually any time you add or
change a client
• It will save you hours of troubleshooting
Amanda for Data Backup (OSCON 2004), Slide 55
amcheck Is The Best Thing Since Sliced Bread
• Cron this script to email you every day before you
leave the office
• Run this script manually any time you touch a
config file
• Run this script manually any time you add or
change a client
• It will save you hours of troubleshooting
• One of the best features of Amanda
Amanda for Data Backup (OSCON 2004), Slide 56
amcheck Can't Do Everything
• am check cannot catch problems that arise after it
runs
• am check doesn't check for everything
• Amanda's other utilities are really good at telling
you why they could not do their job
Amanda for Data Backup (OSCON 2004), Slide 57
Revisiting The amdump Email Report
• Very rarely is the email report as uneventful as
was presented earlier. It typically will include
something like this:
These dumps were to tape CIS-004.
The next tape Amanda expects to use is: CIS-005.
FAILURE AND STRANGE DUMP SUMMARY:
alabama.cis.u /usr lev 1 STRANGE
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 1:00
Run Time (hrs:min) 2:57
etc....
Amanda for Data Backup (OSCON 2004), Slide 58
Revisiting The amdump Email Report (Con't)
• And then later on, it will explain itself....
FAILED AND STRANGE DUMP DETAILS:
/-- alabama.cis.u /usr lev 1 STRANGE
sendbackup: start [alabama.cis.uab.edu:/usr level 1]
sendbackup: info BACKUP=/bin/gtar
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/gtar -f... -
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
? gtar: ./local/majordomo-1.94.5/log/mdlog: file changed as we
read it
| Total bytes written: 207923200 (198MB, 3.1MB/s)
sendbackup: size 203050
sendbackup: end
--------
Amanda is telling us a file was in use as it tried to
grab it. This is usually harmless and constitutes
99% of STRANGE results.
Amanda for Data Backup (OSCON 2004), Slide 59
Revisiting The amdump Email Report
(Con't)
• Sometimes, there are more serious failures
FAILURE AND STRANGE DUMP SUMMARY:
amanda.cis.u //texas/dfs/simnetxpcd lev 0 FAILED [disk
//texas/dfs/simnetxpcd, all estimate failed]
amanda.cis.u //texas/dfs/classfiles lev 0 FAILED [disk
//texas/dfs/classfiles, all estimate failed]
amanda.cis.u //texas/dfs/officefiles lev 0 FAILED [disk
//texas/dfs/officefiles, all estimate failed]
amanda.cis.u //texas/dfs/scripts lev 0 FAILED [disk //texas/dfs/scripts,
all estimate failed]
amanda.cis.u //texas/dfs/home/graduate.private lev 0 FAILED [disk
//texas/dfs/home/graduate.private, all estimate failed]
amanda.cis.u //texas/dfs/home/graduate.public lev 0 FAILED [disk
//texas/dfs/home/graduate.public, all estimate failed]
amanda.cis.u //texas/dfs/home/undergrad.private lev 0 FAILED [disk
//texas/dfs/home/undergrad.private, all estimate failed]
This was because I changed the password to the SAMBA
share that Amanda was using to back up a Windows server.
The email clued me in and the problem was resolved
quickly.
Amanda for Data Backup (OSCON 2004), Slide 60
Handling Tape Failures
• Amanda keeps going and stores as much as
possible in the holding disk
Amanda for Data Backup (OSCON 2004), Slide 61
Handling Tape Failures
• Amanda keeps going and stores as much as
possible in the holding disk
• You can then use am flush to flush data to tape
-bash-2.05b$ amflush CIS
Scanning /dum ps/am anda...
 20040516:found Am anda directory.
 20040517:found Am anda directory.
Multiple Am anda directories,please pick one by letter:
 A.20040516
 B.20040517
Select directories to flush [A..B]:[ALL]
Today is:20040517
Flushing dum ps in 20040516,20040517 using tape changer "chg-scsi".
Expecting tape CIS-015 or a new tape. (The last dum ps were to tape CIS-014)
Are you sure you want to do this [yN]? y
Running in background,you can log offnow.
You'llget m ailwhen am flush is finished.
Amanda for Data Backup (OSCON 2004), Slide 62
Configuring Amanda
What are the prerequisites?
How do I configure my server?
How do I configure my clients?
Amanda for Data Backup (OSCON 2004), Slide 63
Prerequisites
• A server that is mostly idle during the times that
you want to do your backups
• Enough disk space for a suitable holding disk
• GNU tar
• Samba (for Windows clients)
• A large capacity tape drive (typically)
• GNUplot
Amanda for Data Backup (OSCON 2004), Slide 64
Configuration Roadmap
• Set up server
– Create amanda user and assign to group with permission
to use the tape and changer devices (e.g. 'disk' on Linux)
– Gather info on tape and changer devices - mt and mtx
are handy scripts to have here
– Open ports (10080,10082,10083) and set up services
(amanda, amandaidx, amidxtape)
– Configure amanda.conf & changer (e.g. chg-scsi.conf)
• Set up clients
– Create amanda user
– Config ports, services, files, directories
– Allow access (typically .amandahosts)
Amanda for Data Backup (OSCON 2004), Slide 65
Things To Think About Before
Proceeding
• What should my cycle be?
– Once per night? Every three nights?
Amanda for Data Backup (OSCON 2004), Slide 66
Things To Think About Before
Proceeding
• What should my cycle be?
– Once per night? Every three nights?
• If your cycle seems convoluted, try using two:
– One for daily backups, once per night, one week cycle
– One for archives, always full dumps, run manually when
you need it
Amanda for Data Backup (OSCON 2004), Slide 67
Things To Think About Before
Proceeding
• What should my cycle be?
– Once per night? Every three nights?
• If your cycle seems convoluted, try using two:
– One for daily backups, once per night, one week cycle
– One for archives, always full dumps, run manually when
you need it
• Find the right balance for the cycle length
– Short cycles eat up lots of resources doing full dumps
– Long cycles can be a pain to restore from, you might
need 4 tapes for example
Amanda for Data Backup (OSCON 2004), Slide 68
Things To Think About Before
Proceeding
• What should my cycle be?
– Once per night? Every three nights?
• If your cycle seems convoluted, try using two:
– One for daily backups, once per night, one week cycle
– One for archives, always full dumps, run manually when
you need it
• Find the right balance for the cycle length
– Short cycles eat up lots of resources doing full dumps
– Long cycles can be a pain to restore from, you might
need 4 tapes for example
• How many tapes to use?
– How far back into the past do you want to go?
Amanda for Data Backup (OSCON 2004), Slide 69
amanda.conf
• Amanda's main configuration file
• Many options
• Well documented
• Cannot begin to cover everything here, definitely
read the documentation first!
• This is where you define your cycle's parameters
(length, number of runs, number of tapes, etc...)
Amanda for Data Backup (OSCON 2004), Slide 70
amanda.conf - dumptypes
• Different rules for each DLE
define dum ptype root-tar {
global
program "GNUTAR"
com m ent "root partitions dum ped with tar"
com press none
index yes
exclude list "/usr/local/lib/am anda/exclude.gtar"
priority low
}
define dum ptype com p-high {
global
com m ent "very im portant partitions on fast m achines"
com press client best
priority high
}
Amanda for Data Backup (OSCON 2004), Slide 71
disklist
• Defining your DLEs / targets
# File form at is:
#
# hostnam e diskdev dum ptype [spindle [interface]]
# the tape server itself
am anda.cis.uab.edu / com p-root-tar-exclude-holdingdisk
# the directory server
newjersey.cis.uab.edu / com p-root-tar
# the file server
virginia.cis.uab.edu /m z com p-user-tar
virginia.cis.uab.edu /root com p-root-tar
virginia.cis.uab.edu /usr com p-user-tar
virginia.cis.uab.edu /hc com p-root-tar
Amanda for Data Backup (OSCON 2004), Slide 72
Configuring Tape Changers
• Amanda has a generic interface to tape changers
• Tape changer configuration is stored in chg-
scsi.conf
• chg-scsi.conf is one of many changer scripts that
come with Amanda
• This is where you tell Amanda how many drives
you have, which tapes go to which drives, barcode
support, etc...
Amanda for Data Backup (OSCON 2004), Slide 73
Tape Drive Configuration Heads-Up
• There are many choices of changer configuration
scripts (chg-scsi, chg-multi, chg-mtx, chg-manual,
chg-disk, etc...)
• You may have to use one even if you don't have
an actual changer (chg-manual, chg-disk)
• Many options, many chances for confusion
• Please read docs/TAPE.CHANGERS and other
sources of information (listed at end of talk)
Amanda for Data Backup (OSCON 2004), Slide 74
Configuring Your Tape Collection
• Use amlabel to label new tapes
– am labelCIS CIS-000 slot 0
• Then use amtape to build the tapelist
– am tape CIS update
am tape:scanning all30 slots in tape-changer rack:
slot 26:date 20040623 labelCIS-022
etc...
Amanda for Data Backup (OSCON 2004), Slide 75
Special Case Configurations
• Windows Clients
– Use Samba to backup Windows clients
– Configure the Amanda server or other Unix client with
Samba shares
– Then in the disklist point to that server and share
•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
Amanda for Data Backup (OSCON 2004), Slide 76
Special Case Configurations
• Windows Clients
– Use Samba to backup Windows clients
– Configure the Amanda server or other Unix client with
Samba shares
– Then in the disklist point to that server and share
•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
• Firewalls
– --with-tcpportrange=40000,40030 ( something > 1024)
– --with-udpportrange=920,940 (something < 1024)
– or, iptables has amanda support you can enable
Amanda for Data Backup (OSCON 2004), Slide 77
Other Features
Reports
Fine Tuning
More Goodies
Amanda for Data Backup (OSCON 2004), Slide 78
amplot
• Parses amdump logs
• Generates postscript graphs of dumper activity,
holding disk usage, tape usage and bandwidth
activity.
Amanda for Data Backup (OSCON 2004), Slide 79
amplot
Amanda for Data Backup (OSCON 2004), Slide 80
amstatus - Amanda status reports
• Parses logs and reports on dumper efficiency
network free kps: 12120
holding space : 52428800k (100.00%)
dum per0 busy : 0:53:11 ( 49.14%)
dum per1 busy : 0:21:26 ( 19.80%)
dum per2 busy : 1:36:31 ( 89.17%)
dum per3 busy : 0:35:44 ( 33.03%)
taper busy : 0:26:08 ( 24.15%)
0 dum pers busy : 0:07:54 ( 7.31%) not-idle: 0:07:54 (100.00%)
1 dum per busy : 0:47:01 ( 43.44%) not-idle: 0:46:48 ( 99.55%)
no-bandwidth: 0:00:12 ( 0.45%)
2 dum pers busy : 0:18:51 ( 17.42%) not-idle: 0:15:08 ( 80.29%)
no-bandwidth: 0:03:35 ( 19.09%)
start-wait: 0:00:06 ( 0.62%)
3 dum pers busy : 0:15:38 ( 14.45%) no-bandwidth: 0:15:08 ( 96.86%)
client-constrained: 0:00:18 ( 1.95%)
start-wait: 0:00:08 ( 0.89%)
not-idle: 0:00:02 ( 0.30%)
4 dum pers busy : 0:18:48 ( 17.38%) no-dum pers: 0:16:36 ( 88.26%)
not-idle: 0:02:12 ( 11.74%)
This tells me that maybe I ought to increase the
allowable bandwidth for Amanda.
Amanda for Data Backup (OSCON 2004), Slide 81
Other Goodies
• Amanda is a set of over 20 utilities. Some I
haven't even mentioned yet (amoverview,
amverify, amtoc, etc...)
• We've only scratched the surface of the others.
Most have many options, consult the man pages
and other documentation.
Amanda for Data Backup (OSCON 2004), Slide 82
Take Home Lessons
What to tell your boss when you get back to work...
Amanda for Data Backup (OSCON 2004), Slide 83
If You Only Remember One Thing...
• Amanda is very robust, highly scalable, almost
infinitely configurable, and can very likely handle
your data backup situation
Amanda for Data Backup (OSCON 2004), Slide 84
If You Only Remember Two Things...
• Amanda is very robust, highly scalable, almost
infinitely configurable, and can very likely handle
your data backup situation
• When something is wrong, run amcheck first
Amanda for Data Backup (OSCON 2004), Slide 85
If You Only Remember Three Things...
• Amanda is very robust, highly scalable, almost
infinitely configurable, and can very likely handle
your data backup situation
• When something is wrong, run amcheck first
• Once you get Amanda configured, you can trust it,
and move on with your life.
It just works.
Amanda for Data Backup (OSCON 2004), Slide 86
For More Information
• http://www.amanda.org/
• Consult the following docs:
– README
– INSTALL
– docs/* (esp. FAQ and TAPE.CHANGERS)
– example/*
– The AMANDA section in O'Reilly's “Unix Backup and
Recovery”
– man pages
• The user community is wonderful
– Sign up for amanda-users@amanda.org at
http://www.amanda.org/support/mailinglists.php
Amanda for Data Backup (OSCON 2004), Slide 87
My Information
• My email is fran@cis.uab.edu
• This presentation is available at
http://www.cis.uab.edu/fran/
• Much more Amanda information can be found at
the above URL, including:
– A document detailing every step of my configuration
– A complete sample amanda.conf and chg-scsi.conf
– Notes on configuring Samba to backup Windows shares
Amanda for Data Backup (OSCON 2004), Slide 88
My Information
• My email is fran@cis.uab.edu
• This presentation is available at
http://www.cis.uab.edu/fran/
• Much more Amanda information can be found at
the above URL, including:
– A document detailing every step of my configuration
– A complete sample amanda.conf and chg-scsi.conf
– Notes on configuring Samba to backup Windows shares
The End - Thank You!
Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)

Weitere ähnliche Inhalte

Was ist angesagt?

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Amazon Web Services
 
Open Source KMIP Implementation
Open Source KMIP ImplementationOpen Source KMIP Implementation
Open Source KMIP Implementationsedukull
 
DB2 LUW - Backup and Recovery
DB2 LUW - Backup and RecoveryDB2 LUW - Backup and Recovery
DB2 LUW - Backup and Recoveryimranasayed
 
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAAchieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAMarkus Michalewicz
 
Oracle Database 12c Multitenant for Consolidation
Oracle Database 12c Multitenant for ConsolidationOracle Database 12c Multitenant for Consolidation
Oracle Database 12c Multitenant for ConsolidationYudi Herdiana
 
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsOracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsMarkus Michalewicz
 
Data Center Tiers Explained
Data Center Tiers ExplainedData Center Tiers Explained
Data Center Tiers ExplainedData Cave
 
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...Maichino Sepede
 
Demystifying Data Warehouse as a Service
Demystifying Data Warehouse as a ServiceDemystifying Data Warehouse as a Service
Demystifying Data Warehouse as a ServiceSnowflake Computing
 
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...turgaysahtiyan
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesCarole Gunst
 
Network Attached Storage (NAS)
Network Attached Storage (NAS)Network Attached Storage (NAS)
Network Attached Storage (NAS)sandeepgodfather
 
Big Data & Hadoop Tutorial
Big Data & Hadoop TutorialBig Data & Hadoop Tutorial
Big Data & Hadoop TutorialEdureka!
 
IBM Integrated Analytics System ユーザー利用ガイド 20180213
IBM Integrated Analytics System ユーザー利用ガイド 20180213IBM Integrated Analytics System ユーザー利用ガイド 20180213
IBM Integrated Analytics System ユーザー利用ガイド 20180213IBM Analytics Japan
 
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...DataWorks Summit/Hadoop Summit
 
Hyper-Converged Infrastructure: Concepts
Hyper-Converged Infrastructure: ConceptsHyper-Converged Infrastructure: Concepts
Hyper-Converged Infrastructure: ConceptsNick Scuola
 
Presentation v mware virtual san 6.0
Presentation   v mware virtual san 6.0Presentation   v mware virtual san 6.0
Presentation v mware virtual san 6.0solarisyougood
 

Was ist angesagt? (20)

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
 
Open Source KMIP Implementation
Open Source KMIP ImplementationOpen Source KMIP Implementation
Open Source KMIP Implementation
 
DB2 LUW - Backup and Recovery
DB2 LUW - Backup and RecoveryDB2 LUW - Backup and Recovery
DB2 LUW - Backup and Recovery
 
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAAchieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAA
 
Oracle Database 12c Multitenant for Consolidation
Oracle Database 12c Multitenant for ConsolidationOracle Database 12c Multitenant for Consolidation
Oracle Database 12c Multitenant for Consolidation
 
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsOracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
 
Data Center Tiers Explained
Data Center Tiers ExplainedData Center Tiers Explained
Data Center Tiers Explained
 
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...
VxRail Appliance - Modernize your infrastructure and accelerate IT transforma...
 
Backup strategy plan
Backup strategy planBackup strategy plan
Backup strategy plan
 
Snowflake Overview
Snowflake OverviewSnowflake Overview
Snowflake Overview
 
Demystifying Data Warehouse as a Service
Demystifying Data Warehouse as a ServiceDemystifying Data Warehouse as a Service
Demystifying Data Warehouse as a Service
 
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...
High Availability & Disaster Recovery with SQL Server 2012 AlwaysOn Availabil...
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data Pipelines
 
Network Attached Storage (NAS)
Network Attached Storage (NAS)Network Attached Storage (NAS)
Network Attached Storage (NAS)
 
Big Data & Hadoop Tutorial
Big Data & Hadoop TutorialBig Data & Hadoop Tutorial
Big Data & Hadoop Tutorial
 
IBM Integrated Analytics System ユーザー利用ガイド 20180213
IBM Integrated Analytics System ユーザー利用ガイド 20180213IBM Integrated Analytics System ユーザー利用ガイド 20180213
IBM Integrated Analytics System ユーザー利用ガイド 20180213
 
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...
Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...
 
Hyper-Converged Infrastructure: Concepts
Hyper-Converged Infrastructure: ConceptsHyper-Converged Infrastructure: Concepts
Hyper-Converged Infrastructure: Concepts
 
Presentation v mware virtual san 6.0
Presentation   v mware virtual san 6.0Presentation   v mware virtual san 6.0
Presentation v mware virtual san 6.0
 
Database migration
Database migrationDatabase migration
Database migration
 

Andere mochten auch

ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-up
ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-upManageIQ Overview at Management and Orchestration Developer (MODM) Meet-up
ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-upJerome Marc
 
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)Fran Fabrizio
 
Amanda - Presentation
Amanda - PresentationAmanda - Presentation
Amanda - PresentationAmanda Lam
 
Amanda Enterpriseによるクラウドバックアップご紹介
Amanda Enterpriseによるクラウドバックアップご紹介Amanda Enterpriseによるクラウドバックアップご紹介
Amanda Enterpriseによるクラウドバックアップご紹介OSSラボ株式会社
 
Amanda
AmandaAmanda
AmandaNVSBPL
 

Andere mochten auch (6)

AMANDA
AMANDAAMANDA
AMANDA
 
ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-up
ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-upManageIQ Overview at Management and Orchestration Developer (MODM) Meet-up
ManageIQ Overview at Management and Orchestration Developer (MODM) Meet-up
 
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)
Open Source Data Backup, or: How to Sleep Better at Night (OSCON 2005)
 
Amanda - Presentation
Amanda - PresentationAmanda - Presentation
Amanda - Presentation
 
Amanda Enterpriseによるクラウドバックアップご紹介
Amanda Enterpriseによるクラウドバックアップご紹介Amanda Enterpriseによるクラウドバックアップご紹介
Amanda Enterpriseによるクラウドバックアップご紹介
 
Amanda
AmandaAmanda
Amanda
 

Ähnlich wie Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)

High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013Server Density
 
Palmetto Technology Hub - Skill Sharing
Palmetto Technology Hub - Skill Sharing Palmetto Technology Hub - Skill Sharing
Palmetto Technology Hub - Skill Sharing Tina Arnoldi, MA, LPC
 
KoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganKoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganTobias Koprowski
 
Load-testing 101 for Startups with Artillery.io
Load-testing 101 for Startups with Artillery.ioLoad-testing 101 for Startups with Artillery.io
Load-testing 101 for Startups with Artillery.ioHassy Veldstra
 
KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018Tobias Koprowski
 
File storageandbackupsystems
File storageandbackupsystemsFile storageandbackupsystems
File storageandbackupsystemsMicheline LeBlanc
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL databaseDave Stokes
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganTobias Koprowski
 
Dueling duplications RMAN vs Delphix
Dueling duplications RMAN vs DelphixDueling duplications RMAN vs Delphix
Dueling duplications RMAN vs DelphixKyle Hailey
 
D17316 Gc20 App A
D17316 Gc20 App AD17316 Gc20 App A
D17316 Gc20 App Ajaalsuca
 
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganKoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganTobias Koprowski
 
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganKoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganTobias Koprowski
 
Scale by the Bay 2019 Reprogramming the Programmer
Scale by the Bay 2019 Reprogramming the ProgrammerScale by the Bay 2019 Reprogramming the Programmer
Scale by the Bay 2019 Reprogramming the ProgrammerPaul Cleary
 
What the Heck Just Happened?
What the Heck Just Happened?What the Heck Just Happened?
What the Heck Just Happened?Ken Evans
 
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbegan
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbeganKoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbegan
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbeganTobias Koprowski
 
Death of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopDeath of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopErik Riedel
 
Replication, Durability, and Disaster Recovery
Replication, Durability, and Disaster RecoveryReplication, Durability, and Disaster Recovery
Replication, Durability, and Disaster RecoverySteven Francia
 
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierPhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierDave Stokes
 
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...Victor Marmol
 
201657_Patra_pdf
201657_Patra_pdf201657_Patra_pdf
201657_Patra_pdflokinisj
 

Ähnlich wie Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004) (20)

High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013
 
Palmetto Technology Hub - Skill Sharing
Palmetto Technology Hub - Skill Sharing Palmetto Technology Hub - Skill Sharing
Palmetto Technology Hub - Skill Sharing
 
KoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just beganKoprowskiT - SQLBITS X - 2am a disaster just began
KoprowskiT - SQLBITS X - 2am a disaster just began
 
Load-testing 101 for Startups with Artillery.io
Load-testing 101 for Startups with Artillery.ioLoad-testing 101 for Startups with Artillery.io
Load-testing 101 for Startups with Artillery.io
 
KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018KoprowskiT_2AMaDisasterJustBeganAD2018
KoprowskiT_2AMaDisasterJustBeganAD2018
 
File storageandbackupsystems
File storageandbackupsystemsFile storageandbackupsystems
File storageandbackupsystems
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
 
Dueling duplications RMAN vs Delphix
Dueling duplications RMAN vs DelphixDueling duplications RMAN vs Delphix
Dueling duplications RMAN vs Delphix
 
D17316 Gc20 App A
D17316 Gc20 App AD17316 Gc20 App A
D17316 Gc20 App A
 
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBeganKoprowskiT_SPBizConference_2AMaDisasterJustBegan
KoprowskiT_SPBizConference_2AMaDisasterJustBegan
 
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBeganKoprowskiT_SPBizConf_2AMaDisasterJustBegan
KoprowskiT_SPBizConf_2AMaDisasterJustBegan
 
Scale by the Bay 2019 Reprogramming the Programmer
Scale by the Bay 2019 Reprogramming the ProgrammerScale by the Bay 2019 Reprogramming the Programmer
Scale by the Bay 2019 Reprogramming the Programmer
 
What the Heck Just Happened?
What the Heck Just Happened?What the Heck Just Happened?
What the Heck Just Happened?
 
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbegan
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbeganKoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbegan
KoprowskiT_SQLSat219_Kiev_2AM-aDisasterJustbegan
 
Death of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO WorkshopDeath of Disk Panel Session - HEC-FSIO Workshop
Death of Disk Panel Session - HEC-FSIO Workshop
 
Replication, Durability, and Disaster Recovery
Replication, Durability, and Disaster RecoveryReplication, Durability, and Disaster Recovery
Replication, Durability, and Disaster Recovery
 
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierPhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
 
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...
 
201657_Patra_pdf
201657_Patra_pdf201657_Patra_pdf
201657_Patra_pdf
 

Kürzlich hochgeladen

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 

Kürzlich hochgeladen (20)

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 

Amanda for Data Backup, or: How to Sleep Better at Night (OSCON 2004)

  • 1. Amanda for Data Backup, or: How To Sleep Better At Night Fran Fabrizio Senior Systems Administrator Dept. of Computer and Information Sciences University of Alabama at Birmingham fran@cis.uab.edu http://www.cis.uab.edu/fran/ O’Reilly Open Source Convention, July 26–30, 2004
  • 2. Amanda for Data Backup (OSCON 2004), Slide 2 Talk Overview • Introduction to Amanda – What it is, how it thinks • Amanda In Action – Real world examples • Configuring Amanda – Quick look at the config files • Other Features • Take Home Lessons • For More Info
  • 3. Amanda for Data Backup (OSCON 2004), Slide 3 Introduction To Amanda What is Amanda? How does it work? What is its philosophy?
  • 4. Amanda for Data Backup (OSCON 2004), Slide 4 What is Amanda? • Cross-Platform • Scalable • Automated • Flexible • Robust Amanda is the Advanced Maryland Automatic Network Disk Archiver. It has the following features:
  • 5. Amanda for Data Backup (OSCON 2004), Slide 5 What is Amanda? (Cont) • A set of CLI utilities written in C • Its own protocols on top of TCP and UDP • Client and server components that partner to stream and store data for backup and recovery
  • 6. Amanda for Data Backup (OSCON 2004), Slide 6 Holding Disk Tape Drive OS X Client Solaris Client Linux Client Windows Client Amanda Amanda Server amandad Samba amandad amandad Sample Topology
  • 7. Amanda for Data Backup (OSCON 2004), Slide 7 Typical Sequence of Events Amanda Server Amanda Client amdump planner dumper(s) driver taper amandad amandad launches passes schedule spawns writes to disk writes to taper flush to tape requests estimate returns estimate requests backup returns image (many clients serviced in parallel)
  • 8. Amanda for Data Backup (OSCON 2004), Slide 8 How Does It Work? • gtar and dump • Uses standard backup levels • Manages tapes • Balances resources • Supports compression and encryption • Degrades gracefully
  • 9. Amanda for Data Backup (OSCON 2004), Slide 9 Some Amanda Terminology • Dump cycle – How often do you want a full backup? • Disklist Entry / DLE / Target – Something (partition, filesystem) you want to backup
  • 10. Amanda for Data Backup (OSCON 2004), Slide 10 Amanda's Philosophy • “You tell me how often you want a full backup, and I'll worry about everything else” • At least one full backup of each DLE per cycle • You don't get to say when full backups happen • Sounds scary, but usually sufficient – (When it's not, you can hack around it)
  • 11. Amanda for Data Backup (OSCON 2004), Slide 11 Amanda In Action What are some real-world examples? What does typical usage look like? How do I recover a file or partition? What happens when things go wrong?
  • 12. Amanda for Data Backup (OSCON 2004), Slide 12 Real World Examples – CIS @ UAB • Before....
  • 13. Amanda for Data Backup (OSCON 2004), Slide 13 Real World Examples – CIS @ UAB • Before....
  • 14. Amanda for Data Backup (OSCON 2004), Slide 14 Real World Examples – CIS @ UAB • Before....
  • 15. Amanda for Data Backup (OSCON 2004), Slide 15 Real World Examples – CIS @ UAB • Before....
  • 16. Amanda for Data Backup (OSCON 2004), Slide 16 Real World Examples – CIS @ UAB • Before....
  • 17. Amanda for Data Backup (OSCON 2004), Slide 17 Real World Examples – CIS @ UAB • Before.... (And this isn't even the whole dirty truth)
  • 18. Amanda for Data Backup (OSCON 2004), Slide 18 Real World Examples – CIS @ UAB • Before.... I WORRIED CONSTANTLY AND GOT NO SLEE
  • 19. Amanda for Data Backup (OSCON 2004), Slide 19 Real World Examples – CIS @ UAB • Before.... I WORRIED CONSTANTLY AND GOT NO SLEE (I was also exhibiting bizarre behavior...)
  • 20. Amanda for Data Backup (OSCON 2004), Slide 20 Real World Examples – CIS @ UAB • So we bought...
  • 21. Amanda for Data Backup (OSCON 2004), Slide 21 Real World Examples – CIS @ UAB • So we bought... Great! Only one problem...
  • 22. Amanda for Data Backup (OSCON 2004), Slide 22 Real World Examples – CIS @ UAB • This hardware is very expensive
  • 23. Amanda for Data Backup (OSCON 2004), Slide 23 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university...
  • 24. Amanda for Data Backup (OSCON 2004), Slide 24 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university... • ....in Alabama ;-)
  • 25. Amanda for Data Backup (OSCON 2004), Slide 25 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university... • ....in Alabama ;-) • We had no more money
  • 26. Amanda for Data Backup (OSCON 2004), Slide 26 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university... • ....in Alabama ;-) • We had no more money • Amanda saved the day
  • 27. Amanda for Data Backup (OSCON 2004), Slide 27 Real World Examples – CIS @ UAB • This hardware is very expensive • I work for a public university... • ....in Alabama ;-) • We had no more money • Amanda saved the day • Usage at CIS – 43 filesystems/partitions on 14 Linux, Solaris and Windows clients – ~ 60GB of data per night – Fully automated, requiring about 30 minutes of attention in the first half of 2004
  • 28. Amanda for Data Backup (OSCON 2004), Slide 28 Real World Examples - Others • Other Examples – Much Larger • One user wrote about a 700GB nightly dump – Much Smaller • one client systems – Vtape setups • virtual tapes on disk • periodically burn to DVD
  • 29. Amanda for Data Backup (OSCON 2004), Slide 29 Typical Amanda Daily Operation • At 2:30pm, am check utility runs via cron job and informs me via email if there are any problems
  • 30. Amanda for Data Backup (OSCON 2004), Slide 30 Typical Amanda Daily Operation • At 2:30pm, am check utility runs via cron job and informs me via email if there are any problems • At 2:00am, am dum p utility runs to kick off the Amanda backup process
  • 31. Amanda for Data Backup (OSCON 2004), Slide 31 Typical Amanda Daily Operation • At 2:30pm, am check utility runs via cron job and informs me via email if there are any problems • At 2:00am, am dum p utility runs to kick off the Amanda backup process • Sometime in the morning, an email arrives summarizing last night's activity.
  • 32. Amanda for Data Backup (OSCON 2004), Slide 32 Sample amdump Email Output These dumps were to tape CIS-004. The next tape Amanda expects to use is: CIS-005. STATISTICS: Total Full Daily -------- -------- -------- Estimate Time (hrs:min) 1:00 Run Time (hrs:min) 2:57 Dump Time (hrs:min) 3:24 2:02 1:22 Output Size (meg) 20957.1 17678.4 3278.8 Original Size (meg) 38473.8 31408.7 7065.1 Avg Compressed Size (%) 54.5 56.3 46.4 (level:#disks ...) Filesystems Dumped 41 11 30 (1:28 3:1 4:1) Avg Dump Rate (k/s) 1753.8 2474.3 682.5 Tape Time (hrs:min) 0:21 0:15 0:05 Tape Size (meg) 20957.2 17678.4 3278.8 Tape Used (%) 4.3 3.6 0.7 (level:#disks ...) Filesystems Taped 41 11 30 (1:28 3:1 4:1) Avg Tp Write Rate (k/s) 17361.9 19931.6 10242.2 USAGE BY TAPE: Label Time Size % Nb CIS-004 0:21 20957.2 4.3 41
  • 33. Amanda for Data Backup (OSCON 2004), Slide 33 Sample amdump Email Output (Continued)NOTES: planner: Full dump of virginia:/hc promoted from 4 days ahead. planner: Full dump of florida:/var promoted from 4 days ahead. planner: Full dump of alabama:/etc promoted from 4 days ahead. planner: Full dump of florida:/etc promoted from 1 day ahead. planner: Full dump of alabama:/home promoted from 1 day ahead. planner: Full dump of georgia:/home promoted from 4 days ahead. planner: Full dump of illinois:/home promoted from 4 days ahead. planner: Full dump of newyork:/home promoted from 4 days ahead. planner: Full dump of newjersey:/ promoted from 4 days ahead. taper: tape CIS-004 kb 21462176 fm 41 [OK] DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -------------------------- --------------------------------- ------------ missouri.hpcl.c -xport/home 1 24671 13205 53.5 3:42 59.6 0:018848.1 missouri.hpcl.c /export/opt 1 1055 49 4.6 0:44 1.1 0:01 42.8 missouri.hpcl.c /var/mail 1 1538303 828724 53.9 13:171040.4 1:299345.3 nevada.cis.ua / 4 2271760 881675 38.8 5:092850.4 1:0813033.1 ohio.cis.u /home 1 1540 247 16.0 0:01 401.8 0:04 69.7 maine.cis. /home 1 82640 59078 71.5 0:144300.2 0:087671.2 florida.cis.u /etc 0 5200 1613 31.0 0:03 625.5 0:011186.9 florida.cis.u /he 1 8780 614 7.0 0:26 23.4 0:01 512.8 florida.cis.u /hf 1 16440 2066 12.6 2:02 16.9 0:021249.4 etc.... (brought to you by Amanda version 2.4.4p2)
  • 34. Amanda for Data Backup (OSCON 2004), Slide 34 Amanda's planner In Action planner clientA “Please estimate level 0, level 1 and level 2 backups for /home.” Let's assume that last night Amanda performed a level 1 backup of clientA's /home DLE... clientA returns the info “Will promoting this to a level 0 (full) dump lead to more balance over the dump cycle?” “Will we save a significant amount of tape space by going to a level 2 incremental instead of level 1?” Schedules Level 0 dump Schedules Level 2 dump Schedules level 1 dump Yes Yes No No
  • 35. Amanda for Data Backup (OSCON 2004), Slide 35 More about planner • Guarantees one full dump per cycle
  • 36. Amanda for Data Backup (OSCON 2004), Slide 36 More about planner • Guarantees one full dump per cycle • If the tape is too small for the run's data, it will delay some of the dumps in the least disruptive way
  • 37. Amanda for Data Backup (OSCON 2004), Slide 37 More about planner • Guarantees one full dump per cycle • If the tape is too small for the run's data, it will delay some of the dumps in the least disruptive way • Looks at past dumps to determine optimal balancing behavior
  • 38. Amanda for Data Backup (OSCON 2004), Slide 38 More about planner • Guarantees one full dump per cycle • If the tape is too small for the run's data, it will delay some of the dumps in the least disruptive way • Looks at past dumps to determine optimal balancing behavior • Tries to stay as close to level 0 as possible to reduce need to use multiple tapes for restore
  • 39. Amanda for Data Backup (OSCON 2004), Slide 39 Restoring One Or A Few Files Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25. Solution: Use the amrecover utility
  • 40. Amanda for Data Backup (OSCON 2004), Slide 40 Restoring One Or A Few Files [root@alabam a /tm p]# amrecover -C CIS -s amanda.cis.uab.edu -t amanda.cis.uab.edu -d /dev/nst0 AMRECOVER Version 2.4.4p1.Contacting server on am anda.cis.uab.edu ... 220 keep AMANDA index server (2.4.4p2) ready. 200 Access OK Setting restore date to today (2004-06-28) 200 Working date set to 2004-06-28. Scanning /dum ps/am anda... 20040622:found Am anda directory. 200 Config set to CIS. 200 Dum p host set to alabam a.cis.uab.edu. am recover> setdisk /var/spool/mail 200 Disk set to /var/spool/m ail. am recover> setdate 2004-06-25 200 Working date set to 2004-06-25. Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25. Solution: Use the amrecover utility
  • 41. Amanda for Data Backup (OSCON 2004), Slide 41 Restoring One Or A Few Files (Con't) am recover> ls [...] 2004-06-25 brockhw 2004-06-25 brownta 2004-06-25 bryant 2004-06-25 byrdv [...] am recover> add bryant Added /bryant am recover> extract Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu. The following tapes are needed:CIS-024 Restoring files into directory /tm p Continue [?/Y/n]? Y Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu. Load tape CIS-024 now Continue [?/Y/n/s/t]? Y ./bryant am recover> quit 200 Good bye. [root@alabam a /tm p]# ls -l bryant -rw------- 1 bryant disk 14533946 Jun 24 19:10 bryant
  • 42. Amanda for Data Backup (OSCON 2004), Slide 42 Restoring An Entire Backup Target Problem: The disk holding /etc on a critical server has failed Solution: Use the amrestore utility.
  • 43. Amanda for Data Backup (OSCON 2004), Slide 43 Restoring An Entire Backup Target Problem: The disk holding /etc on a critical server has failed Solution: Use the amrestore utility. [root@am anda testrestore]# amadmin CIS info alabama.cis.uab.edu '/etc$' Current info for alabam a.cis.uab.edu /etc: Stats:dum p rates (kps),Full: 481.0,320.0,350.0 Increm ental: 19.0, 13.0, 11.0 com pressed size,Full: 20.6%,20.6%,20.6% Increm ental: 5.4%, 5.4%, 5.4% Dum ps:lev datestm p tape file origK com pK secs 0 20040623 CIS-022 23 18690 3849 8 1 20040628 CIS-027 10 700 38 0 [root@am anda testrestore]# su - amanda -bash-2.05b$ am tape CIS labelCIS-022 am tape:scanning for tape with labelCIS-022 am tape:slot 26:date 20040623 labelCIS-022 (exact labelm atch) am tape:labelCIS-022 is now loaded. -bash-2.05b$ exit
  • 44. Amanda for Data Backup (OSCON 2004), Slide 44 Restoring An Entire Backup Target (Con't) [root@am anda testrestore]# amrestore -p /dev/nst0 alabama.cis.uab.edu /etc > etc.0.tar am restore: 0:skipping start oftape:date 20040623 labelCIS-022 am restore: 1:skipping am anda.cis.uab.edu.__texas_dfs_hom e_undergrad.private.20040623.1 am restore: 2:skipping florida.cis.uab.edu._etc.20040623.1 am restore: 3:skipping alabam a.cis.uab.edu._root.20040623.1 [...] amrestore: 21: skipping missouri.cis.uab.edu._.20040623.0 amrestore: 22: skipping oregon.cis.uab.edu._home.20040623.1 amrestore: 23: restoring alabama.cis.uab.edu._etc.20040623.0 amrestore: 24: reached end of information [root@am anda testrestore]# ls -l total18716 -rw-r--r-- 1 root root 19138560 Jun 28 14:40 etc.0.tar
  • 45. Amanda for Data Backup (OSCON 2004), Slide 45 Other Ways To Restore Files • Amanda not available?
  • 46. Amanda for Data Backup (OSCON 2004), Slide 46 Other Ways To Restore Files • Amanda not available? root@am anda testrestore]# mt -f /dev/nst0 rewind [root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1 AMANDA:TAPESTART DATE 20040623 TAPE CIS-022 1+0 records in 1+0 records out [root@am anda testrestore]# mt -f /dev/nst0 fsf 01 [root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1 AMANDA:FILE 20040623 am anda.cis.uab.edu //texas/dfs/hom e/undergrad.public lev 1 com p .gz program /usr/bin/sm bclient To restore,position tape at start offile and run: dd if=<tape> bs=32k skip=1 |/bin/gzip -dc |usr/bin/sm bclient -f...- 1+0 records in 1+0 records out
  • 47. Amanda for Data Backup (OSCON 2004), Slide 47 When Things Go Wrong • Data backup is a complex interaction between a lot of players, and things -will- go wrong... – Hosts will be down, or away (laptops) – Tapes will go bad – You'll change a password and then forget to tell Amanda you changed it – Your holding disk might be too small
  • 48. Amanda for Data Backup (OSCON 2004), Slide 48 When Things Go Wrong • Data backup is a complex interaction between a lot of players, and things -will- go wrong... – Hosts will be down, or away (laptops) – Tapes will go bad – You'll change a password and then forget to tell Amanda you changed it – Your holding disk might be too small • Amanda tries to be proactive by running am check and giving you time to fix problems
  • 49. Amanda for Data Backup (OSCON 2004), Slide 49 Sample amcheck Problem Report • Host down Amanda Tape Server Host Check ----------------------------- Holding disk /dumps/amanda: 60967688 KB disk space available, that's plenty amcheck-server: slot 5: date 20040520 label CIS-018 (exact label match) NOTE: skipping tape-writable test Tape CIS-018 label ok Server check took 175.627 seconds Amanda Backup Client Hosts Check -------------------------------- WARNING: vermont.cis.uab.edu: selfcheck request timed out. Host down? Client check: 13 hosts checked in 30.210 seconds, 1 problem found (brought to you by Amanda 2.4.4p2)
  • 50. Amanda for Data Backup (OSCON 2004), Slide 50 More amcheck Sample Messages NOTE: index dir / usr/local/etc/amanda/CIS/index/amanda.cis.uab.edu/__texas_dfs_home_undergr ad.private: does not exist ERROR: georgia.cis.uab.edu: [access as amanda not allowed from amanda@amanda] amandahostsauth failed ERROR: virginia.cis.uab.edu: [dir /etc needs 64KB, only has 5KB WARNING: holding disk /dumps/amanda: only 50254708 KB free (52428800 KB requested) amcheck-server: slot 19: rewinding tape: No medium found amcheck-server: slot 19: date 20040330 label CIS-027 (active tape) amcheck-server: fatal slot 20: slot 20 move failed ERROR: label CIS-016 or new tape not found in rack (expecting tape CIS-016 or a new tape) amcheck-server: could not get changer info: could not read result from "/usr/local/libexec/chg-scsi" WARNING: skipping tape test because amdump or amflush seem to be running WARNING: if they are not, you must run amcleanup
  • 51. Amanda for Data Backup (OSCON 2004), Slide 51 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office
  • 52. Amanda for Data Backup (OSCON 2004), Slide 52 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office • Run this script manually any time you touch a config file
  • 53. Amanda for Data Backup (OSCON 2004), Slide 53 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office • Run this script manually any time you touch a config file • Run this script manually any time you add or change a client
  • 54. Amanda for Data Backup (OSCON 2004), Slide 54 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office • Run this script manually any time you touch a config file • Run this script manually any time you add or change a client • It will save you hours of troubleshooting
  • 55. Amanda for Data Backup (OSCON 2004), Slide 55 amcheck Is The Best Thing Since Sliced Bread • Cron this script to email you every day before you leave the office • Run this script manually any time you touch a config file • Run this script manually any time you add or change a client • It will save you hours of troubleshooting • One of the best features of Amanda
  • 56. Amanda for Data Backup (OSCON 2004), Slide 56 amcheck Can't Do Everything • am check cannot catch problems that arise after it runs • am check doesn't check for everything • Amanda's other utilities are really good at telling you why they could not do their job
  • 57. Amanda for Data Backup (OSCON 2004), Slide 57 Revisiting The amdump Email Report • Very rarely is the email report as uneventful as was presented earlier. It typically will include something like this: These dumps were to tape CIS-004. The next tape Amanda expects to use is: CIS-005. FAILURE AND STRANGE DUMP SUMMARY: alabama.cis.u /usr lev 1 STRANGE STATISTICS: Total Full Daily -------- -------- -------- Estimate Time (hrs:min) 1:00 Run Time (hrs:min) 2:57 etc....
  • 58. Amanda for Data Backup (OSCON 2004), Slide 58 Revisiting The amdump Email Report (Con't) • And then later on, it will explain itself.... FAILED AND STRANGE DUMP DETAILS: /-- alabama.cis.u /usr lev 1 STRANGE sendbackup: start [alabama.cis.uab.edu:/usr level 1] sendbackup: info BACKUP=/bin/gtar sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/gtar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: ./local/majordomo-1.94.5/log/mdlog: file changed as we read it | Total bytes written: 207923200 (198MB, 3.1MB/s) sendbackup: size 203050 sendbackup: end -------- Amanda is telling us a file was in use as it tried to grab it. This is usually harmless and constitutes 99% of STRANGE results.
  • 59. Amanda for Data Backup (OSCON 2004), Slide 59 Revisiting The amdump Email Report (Con't) • Sometimes, there are more serious failures FAILURE AND STRANGE DUMP SUMMARY: amanda.cis.u //texas/dfs/simnetxpcd lev 0 FAILED [disk //texas/dfs/simnetxpcd, all estimate failed] amanda.cis.u //texas/dfs/classfiles lev 0 FAILED [disk //texas/dfs/classfiles, all estimate failed] amanda.cis.u //texas/dfs/officefiles lev 0 FAILED [disk //texas/dfs/officefiles, all estimate failed] amanda.cis.u //texas/dfs/scripts lev 0 FAILED [disk //texas/dfs/scripts, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.private lev 0 FAILED [disk //texas/dfs/home/graduate.private, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.public lev 0 FAILED [disk //texas/dfs/home/graduate.public, all estimate failed] amanda.cis.u //texas/dfs/home/undergrad.private lev 0 FAILED [disk //texas/dfs/home/undergrad.private, all estimate failed] This was because I changed the password to the SAMBA share that Amanda was using to back up a Windows server. The email clued me in and the problem was resolved quickly.
  • 60. Amanda for Data Backup (OSCON 2004), Slide 60 Handling Tape Failures • Amanda keeps going and stores as much as possible in the holding disk
  • 61. Amanda for Data Backup (OSCON 2004), Slide 61 Handling Tape Failures • Amanda keeps going and stores as much as possible in the holding disk • You can then use am flush to flush data to tape -bash-2.05b$ amflush CIS Scanning /dum ps/am anda...  20040516:found Am anda directory.  20040517:found Am anda directory. Multiple Am anda directories,please pick one by letter:  A.20040516  B.20040517 Select directories to flush [A..B]:[ALL] Today is:20040517 Flushing dum ps in 20040516,20040517 using tape changer "chg-scsi". Expecting tape CIS-015 or a new tape. (The last dum ps were to tape CIS-014) Are you sure you want to do this [yN]? y Running in background,you can log offnow. You'llget m ailwhen am flush is finished.
  • 62. Amanda for Data Backup (OSCON 2004), Slide 62 Configuring Amanda What are the prerequisites? How do I configure my server? How do I configure my clients?
  • 63. Amanda for Data Backup (OSCON 2004), Slide 63 Prerequisites • A server that is mostly idle during the times that you want to do your backups • Enough disk space for a suitable holding disk • GNU tar • Samba (for Windows clients) • A large capacity tape drive (typically) • GNUplot
  • 64. Amanda for Data Backup (OSCON 2004), Slide 64 Configuration Roadmap • Set up server – Create amanda user and assign to group with permission to use the tape and changer devices (e.g. 'disk' on Linux) – Gather info on tape and changer devices - mt and mtx are handy scripts to have here – Open ports (10080,10082,10083) and set up services (amanda, amandaidx, amidxtape) – Configure amanda.conf & changer (e.g. chg-scsi.conf) • Set up clients – Create amanda user – Config ports, services, files, directories – Allow access (typically .amandahosts)
  • 65. Amanda for Data Backup (OSCON 2004), Slide 65 Things To Think About Before Proceeding • What should my cycle be? – Once per night? Every three nights?
  • 66. Amanda for Data Backup (OSCON 2004), Slide 66 Things To Think About Before Proceeding • What should my cycle be? – Once per night? Every three nights? • If your cycle seems convoluted, try using two: – One for daily backups, once per night, one week cycle – One for archives, always full dumps, run manually when you need it
  • 67. Amanda for Data Backup (OSCON 2004), Slide 67 Things To Think About Before Proceeding • What should my cycle be? – Once per night? Every three nights? • If your cycle seems convoluted, try using two: – One for daily backups, once per night, one week cycle – One for archives, always full dumps, run manually when you need it • Find the right balance for the cycle length – Short cycles eat up lots of resources doing full dumps – Long cycles can be a pain to restore from, you might need 4 tapes for example
  • 68. Amanda for Data Backup (OSCON 2004), Slide 68 Things To Think About Before Proceeding • What should my cycle be? – Once per night? Every three nights? • If your cycle seems convoluted, try using two: – One for daily backups, once per night, one week cycle – One for archives, always full dumps, run manually when you need it • Find the right balance for the cycle length – Short cycles eat up lots of resources doing full dumps – Long cycles can be a pain to restore from, you might need 4 tapes for example • How many tapes to use? – How far back into the past do you want to go?
  • 69. Amanda for Data Backup (OSCON 2004), Slide 69 amanda.conf • Amanda's main configuration file • Many options • Well documented • Cannot begin to cover everything here, definitely read the documentation first! • This is where you define your cycle's parameters (length, number of runs, number of tapes, etc...)
  • 70. Amanda for Data Backup (OSCON 2004), Slide 70 amanda.conf - dumptypes • Different rules for each DLE define dum ptype root-tar { global program "GNUTAR" com m ent "root partitions dum ped with tar" com press none index yes exclude list "/usr/local/lib/am anda/exclude.gtar" priority low } define dum ptype com p-high { global com m ent "very im portant partitions on fast m achines" com press client best priority high }
  • 71. Amanda for Data Backup (OSCON 2004), Slide 71 disklist • Defining your DLEs / targets # File form at is: # # hostnam e diskdev dum ptype [spindle [interface]] # the tape server itself am anda.cis.uab.edu / com p-root-tar-exclude-holdingdisk # the directory server newjersey.cis.uab.edu / com p-root-tar # the file server virginia.cis.uab.edu /m z com p-user-tar virginia.cis.uab.edu /root com p-root-tar virginia.cis.uab.edu /usr com p-user-tar virginia.cis.uab.edu /hc com p-root-tar
  • 72. Amanda for Data Backup (OSCON 2004), Slide 72 Configuring Tape Changers • Amanda has a generic interface to tape changers • Tape changer configuration is stored in chg- scsi.conf • chg-scsi.conf is one of many changer scripts that come with Amanda • This is where you tell Amanda how many drives you have, which tapes go to which drives, barcode support, etc...
  • 73. Amanda for Data Backup (OSCON 2004), Slide 73 Tape Drive Configuration Heads-Up • There are many choices of changer configuration scripts (chg-scsi, chg-multi, chg-mtx, chg-manual, chg-disk, etc...) • You may have to use one even if you don't have an actual changer (chg-manual, chg-disk) • Many options, many chances for confusion • Please read docs/TAPE.CHANGERS and other sources of information (listed at end of talk)
  • 74. Amanda for Data Backup (OSCON 2004), Slide 74 Configuring Your Tape Collection • Use amlabel to label new tapes – am labelCIS CIS-000 slot 0 • Then use amtape to build the tapelist – am tape CIS update am tape:scanning all30 slots in tape-changer rack: slot 26:date 20040623 labelCIS-022 etc...
  • 75. Amanda for Data Backup (OSCON 2004), Slide 75 Special Case Configurations • Windows Clients – Use Samba to backup Windows clients – Configure the Amanda server or other Unix client with Samba shares – Then in the disklist point to that server and share •am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar
  • 76. Amanda for Data Backup (OSCON 2004), Slide 76 Special Case Configurations • Windows Clients – Use Samba to backup Windows clients – Configure the Amanda server or other Unix client with Samba shares – Then in the disklist point to that server and share •am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar • Firewalls – --with-tcpportrange=40000,40030 ( something > 1024) – --with-udpportrange=920,940 (something < 1024) – or, iptables has amanda support you can enable
  • 77. Amanda for Data Backup (OSCON 2004), Slide 77 Other Features Reports Fine Tuning More Goodies
  • 78. Amanda for Data Backup (OSCON 2004), Slide 78 amplot • Parses amdump logs • Generates postscript graphs of dumper activity, holding disk usage, tape usage and bandwidth activity.
  • 79. Amanda for Data Backup (OSCON 2004), Slide 79 amplot
  • 80. Amanda for Data Backup (OSCON 2004), Slide 80 amstatus - Amanda status reports • Parses logs and reports on dumper efficiency network free kps: 12120 holding space : 52428800k (100.00%) dum per0 busy : 0:53:11 ( 49.14%) dum per1 busy : 0:21:26 ( 19.80%) dum per2 busy : 1:36:31 ( 89.17%) dum per3 busy : 0:35:44 ( 33.03%) taper busy : 0:26:08 ( 24.15%) 0 dum pers busy : 0:07:54 ( 7.31%) not-idle: 0:07:54 (100.00%) 1 dum per busy : 0:47:01 ( 43.44%) not-idle: 0:46:48 ( 99.55%) no-bandwidth: 0:00:12 ( 0.45%) 2 dum pers busy : 0:18:51 ( 17.42%) not-idle: 0:15:08 ( 80.29%) no-bandwidth: 0:03:35 ( 19.09%) start-wait: 0:00:06 ( 0.62%) 3 dum pers busy : 0:15:38 ( 14.45%) no-bandwidth: 0:15:08 ( 96.86%) client-constrained: 0:00:18 ( 1.95%) start-wait: 0:00:08 ( 0.89%) not-idle: 0:00:02 ( 0.30%) 4 dum pers busy : 0:18:48 ( 17.38%) no-dum pers: 0:16:36 ( 88.26%) not-idle: 0:02:12 ( 11.74%) This tells me that maybe I ought to increase the allowable bandwidth for Amanda.
  • 81. Amanda for Data Backup (OSCON 2004), Slide 81 Other Goodies • Amanda is a set of over 20 utilities. Some I haven't even mentioned yet (amoverview, amverify, amtoc, etc...) • We've only scratched the surface of the others. Most have many options, consult the man pages and other documentation.
  • 82. Amanda for Data Backup (OSCON 2004), Slide 82 Take Home Lessons What to tell your boss when you get back to work...
  • 83. Amanda for Data Backup (OSCON 2004), Slide 83 If You Only Remember One Thing... • Amanda is very robust, highly scalable, almost infinitely configurable, and can very likely handle your data backup situation
  • 84. Amanda for Data Backup (OSCON 2004), Slide 84 If You Only Remember Two Things... • Amanda is very robust, highly scalable, almost infinitely configurable, and can very likely handle your data backup situation • When something is wrong, run amcheck first
  • 85. Amanda for Data Backup (OSCON 2004), Slide 85 If You Only Remember Three Things... • Amanda is very robust, highly scalable, almost infinitely configurable, and can very likely handle your data backup situation • When something is wrong, run amcheck first • Once you get Amanda configured, you can trust it, and move on with your life. It just works.
  • 86. Amanda for Data Backup (OSCON 2004), Slide 86 For More Information • http://www.amanda.org/ • Consult the following docs: – README – INSTALL – docs/* (esp. FAQ and TAPE.CHANGERS) – example/* – The AMANDA section in O'Reilly's “Unix Backup and Recovery” – man pages • The user community is wonderful – Sign up for amanda-users@amanda.org at http://www.amanda.org/support/mailinglists.php
  • 87. Amanda for Data Backup (OSCON 2004), Slide 87 My Information • My email is fran@cis.uab.edu • This presentation is available at http://www.cis.uab.edu/fran/ • Much more Amanda information can be found at the above URL, including: – A document detailing every step of my configuration – A complete sample amanda.conf and chg-scsi.conf – Notes on configuring Samba to backup Windows shares
  • 88. Amanda for Data Backup (OSCON 2004), Slide 88 My Information • My email is fran@cis.uab.edu • This presentation is available at http://www.cis.uab.edu/fran/ • Much more Amanda information can be found at the above URL, including: – A document detailing every step of my configuration – A complete sample amanda.conf and chg-scsi.conf – Notes on configuring Samba to backup Windows shares The End - Thank You!