Advanced ZFS Integration with FreeBSD

1. Allan Jude: ZFS: Advanced Integration
Personal Background
● Server admin for 16 years
● FreeBSD src/doc committer, Core Team (July 2016-2017)
● Architect of ScaleEngine CDN
● Host of BSDNow.tv podcast
ZFS: What is it?
● built-in volume manager
● pool is thin provisioned to multiple filesystems
● checksums data and metadata
● compression
● COW snapshots, clones
● per fs tunable properties
Snapshots and Clones
● COW means instant snapshots
● blocks refed by snapshot are kept
● no r/w performance impact as no. of snapshot grows
● snapshots take almost no space (~200k metadata) until blocks change
Boot Environments
● Idea from Solaris
● root on ZFS, snapshot before upgrade
● bootloader support enables upgrades without fear
Boot Environment Tooling
● sysadmin/beadm (Shell script, not as good as it could be)
● GSoc 2017: be(8), libbe(3)
○ better management of fs properties for boot integration
○ “deep” boot env support (more below)
○ a proper C library allows integration with pkg(8) and GUIs
What it looks like
The Rest of the Pool
z: root of pool
z/tmp: /tmp
z/audit: not versioned
Example: Laptop
● with boot envs, no need to fear upgrading your laptop before a presentation anymore
Example: Deep Boot Envs
● /usr/src and /usr/obj should be data sets with extra properties for increased performance
● /usr/src and /usr/obj should match running OS in the boot env

2. Deep Boot Environments
newest,{/usr/src,/usr/obj}
cloned,{/usr/src,/usr/obj}
(startup scripts figures out which child datasets to mount)
BEs as Golden Images
● ScaleEngine use boot envs on all servers
● start with stock FreeBSD with security patches
● zfs send
● zfs recv
● temporarily mount to /mnt
● copy select config files
Persist Config Across Firmwares
● Enhanced process further from above
● New /cfg dataset holds persistent configs
● Images symlink those files from /etc, no need to merge /etc anymore
● zfs recv updated image
● zfsbootcfg (ZFS nextboot)
○ if the new image doesn’t work, reboot to old
● upgrade takes seconds
Replace NanoBSD
● Replace nanobsd in appliance with ZFS
● FreeNAS and pfSense have already done so
● Space efficient
● Still get firmware style (whole system image style) updates
● Reliability of ZFS checksums
● Enhanced nextboot: 3 consec. boot failure in <5mins boots rescue system to allow control of appliance
or AWS instance
Encryption Option #1: GELI
● AES-XTS or AES-CBC
● Full block device encryption (key per disk) support in bootloader
● In gptzfsbooot since 2016
● EFI support for booting encrypted pools before end of 2017
● Need console access to enter passphrase
● No keyfile support in bootloader
Encryption Option #2: ZFS Native
● AES-GCM or AES-ICM
● Not all metadata is encrypted, optionally not all datasets, allows datasets to be unmounted and keys
unloaded, data is protected as it is “at rest”
● Scrub & Resilver without keys loaded
● Diff keys for different datasets
● Spring 2018
● (ZFS will checksum both ciphertext and cleartext)
GELI Enhancements
● BSDCan and BSDCam GELI working groups produced enhancements to new metadata structures
● Support USB keys, separate partitions for storing keys
Appliances: Channel Programs
● multiple ZFS admin operations are not atomic and often slow
● New ZCP feature allows short LUA scripts to perform operations with locks held
● Instruction count and memory limited for safety
● Integrated last month. More scripts coming
● See https://www.bsdcan.org/2017/schedule/events/854.en.html
Appliances: Checkpoints (2017Q4)
● upgrade involve more than OS and tools
● checkpoint preserves ALL data (kind of like a snapshot but different)

3. ● can undo operations that snapshot cannot, like destroying or renaming datasets
● if upgrade fails midway, rollback to checkpoint
● preserve checkpoint until upgrade confirmed good
What Would Make ZFS Better For You?
● just came from ZFS development summit
● cross-platform community, very active, interested in features that benefit users
● Would love user input
● FreeBSD foundation & Delphix are partnering to bring RAID-Z vdev expansion
● What do you need?
Near Future Features
● ZSTD Compression (4x-10x faster than gzip, comparable ratio)
● Adaptive Compression (compress as much as possible without slowing system)
● Faster Resilver (sequential I/O)
○ ZFS, with integrated volume manager, can avoid writing whole disk and only write blocks used
○ This enhancement makes resilver use sequential instead of random I/O
● Smarter Resilver (prefetch)
○ combined, 2x-8x faster when replacing disks
● ZIL performance
○ Intent Logs, how ZFS support synchronous writes
● MMP: safe “zpool import” for clusters
○ Lawrence livermore national labs, pool sharing in cluster
● Device Removal (Evacuation)
○ Delphix, move allocation to other disks, doesn’t work for RAID-Z
Further Future Features
● ZIL performance enhancements
● Fast clone deletion
○ Use live instead of dead list
● Spacemap log(faster alloc/free)
○ faster crash recovery, map + log for free space
● ashift policy, Replace 512b with 4Kn disks
○ time dependent geometry: e.g. “all new allocs will be 4k aligned from now on”
● Distributed Parity (DRAID)
○ e.g. 100 disks: broken into 10x10, throughput of a single disk can still be the bottleneck when
resilvering
○ a virtual disk, made up of chunks from all disks, overcomes this
○ virtual spares, minimizes “reduced parity” situations
● VDEV Classes (metadata, block size)
○ e.g. metadata on SSD, data on disk
● 1000x dedup performance using dedup log
○ problem: hash table on disk causes lots of random I/O, write amplification
○ even with successful dedup of data block, metadata still needs to be written
○ solution: make room in hash table by delete blocks with single ref ...
■ (scott: don’t trust this description as I didn’t understand this part)
○ store changes to hash table in log form like with spacelog
○ Proposal looking for funding
ZFSBook.com
● beginner and advanced books
BSDNow.tv
● weekly video podcast
Q&A
● Channel programs implies Lua interpreter in the kernel? yes. Customized for ZFS.
● Would you use boot envs for major FreeBSD version upgrades? A: yes
○ Newer config file format cause trouble on rollback? A: /etc generally lives with the env, config
file formats in FreeBSD are pretty stable

4. ● Memory requirement for ZFS? A: depends on working set, min 512M, ZFS cache max (sysctl)
○ avoid compression feature if memory limited
● HammerFS vs ZFS? A: HammerFS 1&2 still relies on hw RAID, mainly for a cluster fs. ZFS doesn’t
trust disk or RAID hw.
● A tool for the zfs send, recv way of applying updates? A: currently they’re one liners
● Add 2nd disk, cannot boot problem? A: depends on how the disk is connected.
○ Tip: Don’t run completely out of space. Create an empty dataset with 1% of total storage, to
avoid bad things from happening.