4. We needed to be able to keep the backups for at least 30 days.

5. The only way we could do this is by using incremental backups.

6. After ruling out large, expensive proprietary systems, we chose rsync .

8. Because of rsync 's algorithm we do not transmit the full data over the network (except for the first time.)

9. Rsync is very reliable and rarely breaks due to file changes.

11. These simple scripts had no error checking, but they worked very well!

12. We needed to backup Netware servers too, so we used NCP mounting , which was very slow.

13. The Scripts echo "Here we rsync DATA volume on CASADMLX server to /bkup/casadmlx/DATA/ folder on CASARKLX server" rsync -av --link-dest= /bkup/casadmlx/DATA/current root@casadmlx:/media/nss/DATA/ /bkup/casadmlx/DATA/ $date rm /bkup/casadmlx/DATA/current ln -s /bkup/casadmlx/DATA/ $date /bkup/casadmlx/DATA/current rm -r /bkup/casadmlx/DATA/ `date --date='30 days ago' +%F`

14. The Scripts echo "Here we rsync DATA volume on CASADM server to /bkup/casadm/DATA/ folder on CASARKLX server" ncpmount -S casadm -A casadm -U .bkup.cs.users.selkirk -P provo /mnt/casadm/ rsync -av --link-dest= /bkup/casadm/DATA/current /mnt/casadm/DATA/ /bkup/casadm/DATA/ $date rm /bkup/casadm/DATA/current ln -s /bkup/casadm/DATA/ $date /bkup/casadm/DATA/current rm -r /bkup/casadm/DATA/ `date --date='30 days ago' +%F` umount /mnt/casadm/

15. The Scripts echo "Here we rsync INSTRUCTORS volume on FS1 server to /bkup/fs1/INSTRUCTORS/ folder on CASARKLX server" mount.cifs //fs1/INSTRUCTORS /mnt/fs1/ -o user=Administrator,pass=provo rsync -av -e 'ssh -p7774' --link-dest= /bkup/fs1/INSTRUCTORS/current /mnt/fs1/INSTRUCTORS/ /bkup/fs1/INSTRUCTORS/ $date rm /bkup/fs1/INSTRUCTORS/current ln -s /bkup/fs1/INSTRUCTORS/ $date /bkup/fs1/INSTRUCTORA/current rm -r /bkup/fs1/INSTRUCTORS/ `date --date='30 days ago' +%F` umount /mnt/fs1/

17. We wanted to have good error checking, replication, reporting and central control.

18. We also wanted to actually understand what was going on and we wanted a system that would fit our exact needs. So we wrote our own.

20. We decided to implement only the bare-minimum that we needed.

21. The result was a 100-line script that supports custom mount drivers and performs an incremental backup using rsync .

23. Google Code: http://alpine-backup.googlecode.com

24. Our backup directory

25. Configuration files ;; silvoes2lx backup config ;; September 2010 -- Nicholas Paun [APPS] source = "root@silvoes2lx:/media/nss/APPS/" dest = "/bkup/silvoes2lx/APPS" opt = "-e 'ssh -p7774'" delete_after = "+30" [HOME] source = "root@silvoes2lx:/media/nss/HOME/" dest = "/bkup/silvoes2lx/HOME" opt = "-e 'ssh -p7774'" delete_after = "+30"

26. Mount Drivers function mountfs ( $user , $server , $command ) { $return = passthru ( "ssh $user @ $server ' $command stop'" ); return(! $return ); } function umountfs ( $user , $server , $command ) { $return = passthru ( "ssh $user @ $server ' $command start'" ); return(! $return ); }

27. Logging

29. Experimented with rsync daemons

30. We used batch files for replication

31. We ran rsync on cygwin (too slow.)

32. Backed up Windows servers via CIFS mounting

33. We setup Groupwise backups

34. We also started iPrint backups

36. This was our first attempt at offsite backups.

37. It had a drawback of losing sync if files had changed in between batches.

39. The second one uses --read-batch to perform the backup.

40. rsync -av --write-batch=SCRATCH.batch --link-dest=/bkup/caslablx/SCRATCH/current /mnt/caslablx/SCRATCH/ /bkup/caslablx/SCRATCH/$date

41. ssh n10arklx rsync -av --read-batch=- --link-dest=/bkup/caslablx/SCRATCH/current /bkup/caslablx/SCRATCH/$date <SCRATCH.batch

43. The QNAPs are NASs running some mystery meat Linux, that was the source of many headaches.

44. Initially, we added more configuration files that would copy the backups from the main backup server to the offsite one, but this grew quite complicated.

45. Selkirk College Backup System

47. But, when a QNAP reads the configuration files, it would run in slave mode and pull the backups from the backup server, instead of the live server.

48. This change removed the need for half of our configuration files.

49. Demo: No-change Syncing

50. Demo: No-change Syncing

52. This means that we can apply rsync 's deduplication ( --link-dest ) to GroupWise too.

53. We save 90-95% disk space and it is 10 times faster compared to a full backup.

55. To snapshot NSS volumes we used Dean Giles' perl scripts (found on Cool Solutions).

56. If we ran GroupWise on an LVM volume, we could use LVM snapshots instead.

57. GroupWise Mount Driver function mountfs ( $user , $server , $command ) { $return = passthru ( &quot;snapup.sh&quot; ); return(! $return ); } function umountfs ( $user , $server , $command ) { $return = passthru ( &quot;snapdn.sh&quot; ); return(! $return ); }

59. Demo: GroupWise PO Restore

60. Demo: GroupWise PO Restore

61. Demo: GroupWise PO Restore

63. So we added a MySQL agent to our tools.

64. To gain maximum space savings, we split up the SQL files into one file per table.

65. Now we can use rsync 's --link-dest feature on our DBs too.

67. Demo: Drupal DB Restore

68. Demo: Drupal DB Restore

69. Demo: Drupal DB Restore

70. Demo: Drupal DB Restore

72. We had never seen an Oracle database before, so we did some searching and figured out how to use expdp.

73. First we logged in to the database and set up a directory handle:

74. CREATE DIRECTORY sgrcdev AS '/g/sgrcdev' ;

76. expdp system/$passwd@$db directory=$db full=y dumpfile=$db.dmp logfile=$db.log

77. … And then, we just copy the backups over and reset dump directory.

78. Oracle Backups

80. Now, we can control all of our backups (FS, GroupWise and Database) from one place, using one lightweight tool.

81. We save a lot of disk space by using rsync .

82. Most importantly, we achieved peace of mind.