TL;DR ZFS as a root device on top of LUKS
I encrypt my devices. This saves hours of changing passwords in case a hard disk is lost. Additionally, I don’t have to think much about which data I save: passwords, signatures and photos.
These are my notes in case I need to set up the layout again.
I’m using two SSDs, Sabayon, cryptsetup, genkernel, grub2 and EFI.
1. Hard disk setup:
I’ve the following partition layout:
fdisk -l /dev/sd{a,b}
Disk /dev/sda: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1E98A553-FD92-45B0-AAAD-9B18F113173F
Device Start End Sectors Size Type
/dev/sda1 2048 411647 409600 200M EFI System
/dev/sda2 411648 1435647 1024000 500M Linux filesystem
/dev/sda3 1435648 250068991 248633344 118.6G Linux filesystem
Disk /dev/sdb: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1E98A553-FD92-45B0-AAAD-9B18F113173F
Device Start End Sectors Size Type
/dev/sdb1 2048 411647 409600 200M EFI System
/dev/sdb2 411648 1435647 1024000 500M Linux filesystem
/dev/sdb3 1435648 250068991 248633344 118.6G Linux filesystem
The 200MiB EFI Partition makes booting with grub easier. Especially with larger HDDs and GPT. The second disk holds a backup.
sd{a,b}2 is formatted in ext2. I use it for grub2 and kernel images. I didn’t set up MD devices.
sd{a,b}3 is a LUKS device. These are formatted using cryptsetup.
2. LUKS
Do a benchmark first and check the performance of your preferred cipher:
cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 819200 iterations per second
PBKDF2-sha256 493215 iterations per second
PBKDF2-sha512 411529 iterations per second
PBKDF2-ripemd160 1048576 iterations per second
PBKDF2-whirlpool 189959 iterations per second
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 516.3 MiB/s 2238.0 MiB/s
serpent-cbc 128b 68.2 MiB/s 446.6 MiB/s
twofish-cbc 128b 142.7 MiB/s 281.5 MiB/s
aes-cbc 256b 386.6 MiB/s 1713.1 MiB/s
serpent-cbc 256b 68.2 MiB/s 433.5 MiB/s
twofish-cbc 256b 146.8 MiB/s 278.2 MiB/s
aes-xts 256b 1944.4 MiB/s 1948.6 MiB/s
serpent-xts 256b 445.5 MiB/s 432.7 MiB/s
twofish-xts 256b 275.3 MiB/s 281.4 MiB/s
aes-xts 512b 1448.5 MiB/s 1422.2 MiB/s
serpent-xts 512b 449.1 MiB/s 434.0 MiB/s
twofish-xts 512b 272.5 MiB/s 270.6 MiB/s
aesni_intel is active on my machine. So will go with aes-xts. Beware that the xts-version effectively halves your key length.
This creates a new LUKS container with SHA512 instead of the default SHA1:
cryptsetup luksFormat /dev/sdX3 -c aes-xts-plain64:sha512 -s 512
Do this for both devices:
cryptsetup luksOpen /dev/sdX3 cryptX
3. ZFS setup
Setting up ZFS is not in the scope of this article. It’s actually quite easy after you get used to the auto mounting. You can disable it at import with the “-N” parameter or by prepending some path, e.g. “-R /mnt/gentoo”.
This is my final setup using two crypt devices in stripping mode:
zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 236G 112G 124G – 34% 47% 1.00x ONLINE –zpool status
pool: rpool
state: ONLINE
scan: scrub repaired 0 in 0h6m with 0 errors on Sat Aug 22 21:54:00 2015
config:NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
crypt1 ONLINE 0 0 0
crypt2 ONLINE 0 0 0errors: No known data errors
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 112G 117G 96K none
rpool/root 112G 117G 96K none
rpool/root/home 89.8G 117G 72.2G /home
rpool/root/rootfs 21.8G 117G 14.0G /
rpool/root/tmp 1.54M 117G 112K /tmp
Sometimes /tmp isn’t mounted properly because the init scripts already created files there. This can be fixed by ordering zfs to mount over non-empty directories:
zfs set overlay=on rpool/root/tmp
4. The kernel and opening the LUKS container
This part was difficult. sys-kernel/genkernel-next-63 doesn’t work. The system just stops after unlocking the LUKS containers.
I used this line with genkernel-9999
genkernel –makeopts=”-j5″ –no-clean –no-mrproper –zfs –firmware –e2fsprogs –busybox –install –gpg –luks –linuxrc=/root/linuxrc all
Copy the system linuxrc into /root and edit it:
— /usr/share/genkernel/defaults/linuxrc 2015-08-22 22:58:25.269515564 +0200
+++ /root/linuxrc 2015-08-23 11:31:23.801869315 +0200
@@ -391,6 +391,32 @@
# Setup md device nodes if they dont exist
setup_md_device+
+
+# source: http://linux.arantius.com/installing-gentoo-into-a-luks-encrypted-zfs-root
+# setups two crypt devices
+CRYPT_NUM=0
+luks_open() {
+ DISK=”$1″
+ PASSPHRASE=”$2″
+ cryptsetup isLuks “$DISK” || return
+ CRYPT_NUM=$(( CRYPT_NUM + 1))
+ good_msg “Opening $DISK as crypt${CRYPT_NUM} …”
+ # –allow-discards has serious security implications, see
+ # http://asalor.blogspot.de/2011/08/trim-dm-crypt-problems.html
+ echo “$PASSPHRASE” | cryptsetup luksOpen –allow-discards “$DISK” crypt$CRYPT_NUM
+ test_success “luksOpen $DISK”
+}
+echo -n “Please enter LUKS passphrase: ”
+read -s PASSPHRASE
+echo
+for D in /dev/sda3 /dev/sdb3; do
+ luks_open “$D” “$PASSPHRASE”
+done
+PASSPHRASE=”overwriteStringInMemory”
+unset CRYPT_NUM D PASSPHRASE
+
+
# Scan volumes
startVolumes
The slightly changed code is from this homepage. Thanks!
Use this line to upgrade to a new kernel:
genkernel –makeopts=”-j3″ –no-clean –no-mrproper –zfs –firmware –e2fsprogs –busybox –install –gpg –luks –linuxrc=/root/linuxrc –callback=”emerge spl zfs zfs-kmod” all
I suspecect that sometimes the initrd is not regenerated. Use “initramfs” in place of “all” as genkernel parameter.
5. GRUB2
I use this configuration:
cat /etc/default/sabayon-grub
# custom zfs command line
GRUB_CMDLINE_LINUX=”vconsole.keymap=de real_root=ZFS=rpool/root/rootfs dozfs=force “
I left out the “crypt_root=XXX” part on purpose. This way the patched code fragment opens the crypt devices and the stock startLuks function is not used.
This is what my consoleargs in /boot/grub/grub.cfg look like:
linux /kernel-genkernel-x86_64-4.1.5-gentoo root=/dev/mapper/root ro vconsole.keymap=de real_root=ZFS=rpool/root/rootfs dozfs=force noquiet acpi_osi=
6. Conclusion
I really like ZFS. It holds every promise BTRFS ever made. You can easily enable compression with
zfs set compression=on rpool
Only trim/discard is missing in the Linux version. There is a patched version. But this enables an experimental feature and could make mounting the zpool with a livecd impossible.
Making backups became much easier with ZFS. You can send the whole filesystem including metadata:
zfs snapshot -r rpool@2015-08-22
zfs send -R rpool@2015-08-22 | pv | zfs receive -F backup/zen/zfs