Robust Upgrade on Ubuntu, using ZFS and Containers
Introduction
Recently I stumbled across a twitter post, highlighting the benefits of ZFS Boot Environments; see here. Next in that thread, it states:
Unfortunately, #linux has little to offer with the same functionality as #ZFS, especially with Red Hat abandoning #BTRFS
See here. I took this as an implication that it’s not possible to implement a solution similar to ZFS Boot Environments on Linux, which I don’t completely agree with.
The following provides some details about how ZFS Boot Environments could be implemented on Ubuntu, if somebody wanted to do it (we’re doing a variation of this at Delphix).
Step 1: Create an Ubuntu system using ZFS as the root filesystem
Before we can begin, we first need to build an Ubuntu system such that it uses ZFS for the root filesystem; this means using OpenZFS on Linux, which is readily available and supported on the most recent Ubuntu releases.
There’s many different ways to go about doing this, and there’s also excellent tutorials such as this one, so I’ll leave this as an exercise for the reader (we currently use this script at Delphix).
As an example, I have the following Ubuntu 18.04 system, with its root filesystem configured on ZFS:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
$ mount | awk '$3 == "/" {print $0}'
rpool/ROOT/delphix.9aTvgfL on / type zfs (rw,relatime,xattr,posixacl)
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 21.2G 46.1G 64K none
rpool/ROOT 14.7G 46.1G 65K none
rpool/ROOT/delphix.9aTvgfL 14.7G 46.1G 14.7G /
rpool/crashdump 65.5K 17.4G 65.5K /var/crash
rpool/home 6.50G 46.1G 6.50G /export/home
rpool/update 68K 15.0G 68K /var/dlpx-update
Step 2: Create container from clone of root filesytem
Once this is in place, it’s relatively straight forward to create a container from a root filesystem clone, using systemd-nspawn.
Create the clone, mounted in
/var/lib/machines
:# mktemp -d -p /var/lib/machines -t delphix.XXXXXXX /var/lib/machines/delphix.DEepQDm # zfs snapshot rpool/ROOT/delphix.9aTvgfL@delphix.DEepQDm # zfs clone -o mountpoint=/var/lib/machines/delphix.DEepQDm \ > rpool/ROOT/delphix.9aTvgfL@delphix.DEepQDm rpool/ROOT/delphix.DEepQDm # zfs list -o mountpoint,mounted,origin rpool/ROOT/delphix.DEepQDm MOUNTPOINT MOUNTED ORIGIN /var/lib/machines/delphix.DEepQDm yes rpool/ROOT/delphix.9aTvgfL@delphix.DEepQDm
Install the
systemd-container
package, which containssystemd-nspawn
:# apt-get install -y systemd-container
Override default container configuration:
# mkdir -p /etc/systemd/nspawn # cat >/etc/systemd/nspawn/delphix.DEepQDm.nspawn <<EOF > [Exec] > PrivateUsers=no > [Files] > PrivateUsersChown=no > EOF
Start the container:
# systemd-nspawn --capability=all --boot --machine delphix.DEepQDm
At this point, a container should be running on the system, using the cloned root filesystem as it’s root filesystem. Let’s verify this:
# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
delphix.DEepQDm container systemd-nspawn ubuntu 18.04 -
# machinectl status delphix.DEepQDm | head -n10
delphix.DEepQDm(5c5c6f1ae47a4ff29024aa845091ae8c)
Since: Thu 2018-11-29 23:48:37 UTC; 38s ago
Leader: 6597 (systemd)
Service: systemd-nspawn; class container
Root: /var/lib/machines/delphix.DEepQDm
OS: Ubuntu 18.04.1 LTS
Unit: machine-delphix.DEepQDm.scope
├─init.scope
│ └─6597 /lib/systemd/systemd
└─system.slice
We can even run commands in the container like so:
# systemd-run --quiet --machine delphix.DEepQDm --pipe /usr/bin/env systemd-detect-virt
systemd-nspawn
# systemd-run --quiet --machine delphix.DEepQDm --pty /bin/bash
(container)# exit
exit
Step 3: Upgrade the container
Now that we have a container running based on a clone of the host’s root filesystem, we can test out upgrade, without ever modifying the host’s root filesystem.
In this example, the host is running Ubuntu’s “bionic” release, so the container is also running that release. Next, we’re going to upgrade the container to Ubuntu’s latest release, which is “cosmic”.
Check the container’s Ubuntu version before the upgrade:
# systemd-run --quiet --machine delphix.DEepQDm --pipe /usr/bin/lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic
Perform the upgrade of the container:
# systemd-run --quiet --machine delphix.DEepQDm --pty /bin/bash (container)# sed -i 's/bionic/cosmic/g' /etc/apt/sources.list (container)# apt-get update (container)# apt-get dist-upgrade -y (container)# exit
Check the container’s Ubuntu version after the upgrade:
# systemd-run --quiet --machine delphix.DEepQDm --pipe /usr/bin/lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.10 Release: 18.10 Codename: cosmic
Verify the host’s Ubuntu version remains the same:
# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic
Step 4: Use the container’s (upgraded) root filesystem to boot the host
So far, we’ve created a container using a clone of the original root filesystem, and have upgrade that container (and the associated cloned filesystem) to the new Ubuntu version. The last step is to update the host’s bootloader configuration, such that it’ll use this upgraded root filesystem as it’s root filesystem, the next time the host boots.
Stop the container (if it’s still running):
# machinectl poweroff delphix.DEepQDm
Bind mount directories required to update bootloader:
# for dir in /proc /sys /dev; do > mount --rbind "$dir" "/var/lib/machines/delphix.DEepQDm$dir" > mount --make-rslave "/var/lib/machines/delphix.DEepQDm$dir" done
Update bootloader:
# chroot /var/lib/machines/delphix.DEepQDm update-grub # chroot /var/lib/machines/delphix.DEepQDm grub-install /dev/sda
Remove mounts created in (2):
# for dir in /proc /sys /dev; do > umount -R "/var/lib/machines/delphix.DEepQDm$dir" done
Unmount new root dataset, configure it, reboot:
# zfs umount rpool/ROOT/delphix.DEepQDm # zfs set canmount=noauto rpool/ROOT/delphix.DEepQDm # zfs set mountpoint=/ rpool/ROOT/delphix.DEepQDm # reboot
After the system reboots, it’ll be running the root filesystem that was previously upgraded by the container; i.e. the host will be running the Ubuntu “cosmic” release.