Avery's presentation about bup at LinuxFest Northwest 2011 in Bellingham, WA.

1. bup: the git-based backup system Avery Pennarun 2011 04 30

3. Including huge VM disk images (files >100GB)

4. Lots of separate files (500k or more)

5. Calculate/store incrementals efficiently

6. Create backups in O(n), where n = number of changed bytes

7. Incremental backup direct to a remote computer (no local copy)

8. ...and don't forget metadata 5

11. Dedupe between different client machines

12. O(log N) seek times to any part of any file

13. You can mount your backup history as a filesystem or browse it as a web page 8

14. The Design 8

16. Easily deduplicates between identical files and trees

17. Debug my code using git commands

18. Someone already thought about the repo format (packs, idxes)

19. Three-level “work tree” vs “index” vs “commit” 11

21. git bigfiles fork “solves” the problem by just never deltifying large objects: lame

22. zlib window size is very small: lousy compression on VM images -- 13

25. Duplicated data >64k apart can't be compressed

29. He wrote a (readable) Ph.D. thesis about it

30. bup uses a variant of the rsync algorithm to -- 19

33. But any n-byte subsequence might be the match

34. Searching naively is about O(n^2) ... ouch.

35. So we use a rolling checksum instead 21

38. Requires a special “rollable” checksum (adler32) 23

40. Changing a byte early in a file changes the compression dictionary, so the rest of the file is different

41. --rsyncable resets the compression dictionary whenever low bits of adler32 == 0

42. Fraction of a percent overhead on file size

43. But now your gzip files are rsyncable! 26

47. Inserting/deleting bytes changes at most two chunks! 28

50. If low 21 bits are 1, end a superduperchunk

52. Inserting/deleting bytes changes at most 2*log(n) chunks! 30

54. Compresses most VM images more (and faster) than gzip

55. Works well on binary and text files

56. Don't need to teach it about file formats

57. Diff huge files in about O(log n) time

58. Seek to any offset in a file in O(log n) time 32

60. x 20 bytes per SHA1: 2.4GB

61. Divided into 2GB packs: 500 .idx files of 5MB each

64. Larger initial lookup table to immediately narrow search to the last 7 bits

65. log(2.4GB) = 31 bits

66. 24-bit lookup table * 4 bytes per entry: 64 MB

67. Adding a new chunk always only touches two pages 35

69. Storing 20 bytes per hash is wasteful; even 48 bits would be unique

71. Create a big hash table

72. Hash each block several ways

73. Gives false positives, never false negatives

74. Can update incrementally 36

77. bup ftp (command-line browsing)

78. bup web

79. bup fuse (mount backups as a filesystem) 36

82. 'bup restore' that updates the index like 'git checkout' does

83. Distributed filesystem: bittorrent with a smarter data structure 39

84. Questions? 40