Here’s the detail behind the video. If you decide to upgrade your Incus Server OS (and I suggest not to do so right now) then you may be in for a lot of trouble like me.
This is a warning about Upgrading your Incus Server OS to Ubuntu 24.04 LTS if you have nested docker containers inside your Incus containers. Ubuntu 24.04 introduces zfs v2.2 and docker has not adapted!
If you upgrade the host OS of your incus server from Ubuntu 22.04 to either Ubuntu 23.10 or Ubuntu 24.04, docker applications nested inside of your incus containers will not be running after the upgrade.
According to Stéphane Graber who is the chief maintainer of Incus, Ubuntu 24.04 LTS updates to ZFS 2.2 which now natively supports VFS idmap shifting, meaning that all the data in /var/lib/incus/containers is stored unshifted (as seen in the container) rather than having all uid/gid altered to match the container’s namespace.
This means that any current incus containers you have before the upgrade and all the incus backups (exports) you have prior to the upgrade will essentially be missing all your docker vfs storage.
Note that I am talking about the Incus server OS version and not the OS version of your incus containers. You can use any Ubuntu OS variant in your incus containers regardless and that is not an issue.
This might sound like an incus problem or an Ubuntu problem. In reality, it appears that changes in zfs 2.2 have affected the docker vfs storage because of this remapping. It should have been something that a docker upgrade that should have happened at the same time that Ubuntu was upgraded. This simply flowed down to incus containers that have nested docker instances.
Although docker persistent volume information that you store in your docker mapped volumes for applications is intact, the docker vfs storage in incus containers loses its mapping and therefore is trashed.
That’s hard to understand since you are thinking “well I have an Ubuntu 22.04 host, but each of my docker containers is nested and isolated inside of an incus container”. Well, so remember when I said that Incus/LXD containers are lean and fast because they share the host kernel?
File management is a part of that host and changing out zfs was a bit like pulling the rug out from under unsuspecting containers. Docker didn’t fair well in all that because it, like incus & LXD share these same host elements.
I was sure to create backups of all my containers:
incus export
However, I found that importing the backups was not a solution because the incus containers with nested docker still used the older storage scheme for vfs data and the mapping in zfs 2.2 changed the handling of the id mapping apparently.
My problematic NginX Proxy Manager (NPM) container I had 10 backups of and although they all restored, none worked.
The reason is that with docker vfs data missing/lacking, when you start your docker application it performs a:
docker compose pull
and restores the containers to their previous glory.
You would think that makes it all good. Unfortunately, some apps like Rocketchat and NPM don’t just look at their non-volatile data in the mapped volume folders, they also very much care about the state of the vfs data. Remember that’s gone!
Given that, some apps like NPM just tend to try and start from scratch by pulling their container overlays, but they also try to start by thinking they are newly installed but seeing existing data in their mapped volume folders. That ends up spelling disaster and that never got my NPM running even from scratch.
So, I bit the bullet by reinstalling NPM completely from scratch in a brand new incus container with freshly installed docker. This basically worked… well sorta…
I did this “from scratch install” 17 times. I have 43 reverse proxy entries. Right about the time I got to entry 24, the program either froze, logged me off or restarted. I still don’t know why. Remember, I spent 28 hours straight trying to fix this and give you guys a good answer.
I do want to tell you that Rocketchat had an initial problem, but was easy to get around with a “docker build”. I ended up hosting NPM in an Incus VM. The difference with an incus VM is that it doesn’t share any host kernel elements and instead does a software virtual kernel. I am still not sure why NPM was special, but that is what I did. Should you decide to upgrade (in retrospect, I wouldn’t) then pack lots of Mountain Dew and extra patience.
With that in mind, I did upgrade both my incus application server OS to Ubuntu 24.04 for the purposes of the channel. I nearly peformed ritualistic Japanese Seppuku unknowingly by trying this. Don’t try this at home.
In general, the recommended way to upgrade any Ubuntu is to first upgrade your repositories and to take all updates.
sudo apt update && sudo apt upgrade -y
After that, it is a best pratice to reboot:
sudo reboot now
Just to make sure that your hardware virtualization is running properly in order to be able to run incus virtual machines, install the cpu checker:
sudo apt install cpu-checker
To check and make sure that you have your BIOS switch for hardware virtualization turned on and that you have the software virtualization requires, use this command:
kvm-ok
The command to upgrade Ubuntu is:
sudo do-release-upgrade
As of June 2024, there is no direct path the upgrade Ubuntu from 22.04 LTS to 24.04 LTS and so you will see the following message.
We will have to change the system settings to be able to take the minor release version upgrade to Ubuntu 23.10 first by editing the file:
sudo nano /etc/update-manager/release-upgrades
When you edit this file, change “prompt=lts” to “prompt=normal”:
Do a CTRL O and Enter to save the file and a CTRL X to exit the nano editor.
Execute the update command again:
sudo do-release-upgrade
The update process will continue and if you want to see the detail, reference the notes for Upgrading Incus Container OS because the process is the same.
After the upgrade ends, I opt to delete the obsolete files and then I reboot.
I am now at Ubuntu 23.10.
I now did another:
sudo do-release-upgrade
and the system upgrades from Ubuntu 23.10 to Ubuntu 24.04 LTS.
After the upgrade completes, I am at Ubuntu 24.04 LTS Noble Numbat.
After the reboot, note in LXConsole that in the driver section there is no listing for QEMU and that means that Incus Virtual Machines cannot be supported. If you are not running LXConsole, see my LXConsole Web Interface for Incus notes file.
Since the QEMU drivers are not showing up, that means that my Incus Virtual Machines are not running.
Let’s fix that. Move to the root account and move to the folder where the repositories are stored.
sudo su
cd /etc/apt/sources.list.d
We need to re-enable the Incus Zabbly repository after the upgrade of Ubuntu which disables third party repositories.
nano zabbly-incus-stable.sources
We want to change the “enabled” field to “yes” and change the “Suites” field from “Jammy” to “Noble” as follows:
Do a CTRL O and Enter and a CTRL X to exit the nano editor after making the changes.
Update your repositories:
sudo apt update
Take the upgrade to the latest version of incus to add back in the missing QEMU support.
sudo apt upgrade -y
There are three Incus upgrades as a result of the Ubuntu upgrade.
LXConsole will now list the QEMU drivers in addition to the lxc container drivers which are the underlying infrastructure for both LXD and Incus.
Now all three of my Incus Virtual Machines are now running.
I hope this information helps you all. My suggestion for now would be to not upgrade your Incus Server Host past Ubuntu 22.04 LTS if you have incus containers with nested docker instances in them.
As an example of what to expect if you do the upgrade (despite my multiple warnings), I connected into my “netboot” container below which has nested docker after the upgrade.
As you can see, the app is not running because of the issue I mentioned at the start of these notes.
You can generally fix this by:
docker compose up -d
That will end up pulling a new version of the container overlays which has vanished. That’s easy with simpler apps. More complex apps that have “docker build” processes or docker apps that seem more context sensitive like NPM may have problems.
So, be aware that you will have to perform new backups of your incus containers that have nested docker applications after the Ubuntu 24.04 LTS upgrade on your incus server.