This presentation provides one possible solution for the problem with incus containers that have nested docker applications losing data when upgrading the Incus Server host OS to Ubuntu 24.04.
In my last tutorial “WARNING: Ubuntu 24.04 Hates Incus” I showed how to upgrade your incus server OS from Ubuntu 22.04 to Ubuntu 24.04. Howver, I encountered problems whereby data was missing from the /var/lib/docker vfs area of incus containers with nested docker after the upgrade and so I recommended not upgrading your Incus Server OS to Ubuntu 24.04.
In my case, I did upgrade both of my production incus server hosts to Ubuntu 24.04 and I discovered that any containers that did not have nested docker applications ran just fine. The containers with nested docker in them (regardless of container OS), required:
docker compose up -d
to pull the container components again and restart the app. This didn’t work with all containers with nested docker. Applications that build their container components need to:
docker build
according to the application installation instructions.
In any case, all docker persistent mapped volume folders were intact.
Not all of my applications recovered gracefully and required additional work and as close as I can determine, this is due to changes in the Ubuntu 24.04 kernel to support zfs 2.2. Because of the change in the Ubuntu 24.04 kernel, changes occurred in what is supported at the VFS file system level. Docker, by default uses the VFS filesystem with its overlay2 data type. The issue with container data structures vanishing was caused by changes in the way that idmap shifting was occurring.
New incus containers with nested docker created after the Ubuntu 24.04 upgrade should not experience this problem.
Also realize that incus exports (backups) created prior to the upgrade of the incus server OS to Ubuntu 24.04 are invalid because when I imported old backups, the /var/lib/docker info was also missing. So, after the upgrade, you will need new backups.
One possible solution to this issue was proposed by Stéphane Graber who is the chief maintainer of Incus. The idea is to move all of the docker data from the container rootfs to a dedicated volume in the default storage pool that won’t be affected when the Incus server is updated from Ubuntu 22.04 to Ubuntu 24.04.
So, the procedure that follows would be performed prior to upgrading your incus server host OS to Ubuntu 24.04.
The idea behind this is to avoid idmap shifting during the upgrade of your incus server to Ubuntu 24.04.
Create storage volumes for the container(s) that have nested docker applications. I am calling my storage volume “docker-cont1” with the idea that this is a storage volume for a container named “cont1”. Realize that storage volumes are separate storage areas outside of the root file system of the container that use space in your storage pool.
Watch Fix Docker Issues to understand the storage in the vfs space that docker uses and size it accordingly for your application.
First create a volume:
incus storage volume create default docker-cont1 size=20GB
Create a virtual device mapping (I arbitrarily call the device “docker”) to my container “cont-1” and mount it to /mnt/docker in the container.
incus config device add cont-1 docker pool=default source=docker-cont1 path=/mnt/docker
Stop the docker daemon in container “cont-1”, since we don’t want any locked files:
incus exec cont-1 – systemctl stop docker
Copy the docker files to the mount point which is pointing to the volume:
incus exec cont-1 – sh -c “mv /var/lib/docker/* /mnt/docker/”
Stop the container:
incus stop cont-1
Change the mounted location of the virtual device for the volume to point to the copied data:
incus config set cont-1 docker path=/var/lib/docker
Start the container again:
incus start cont-1
This moves all of the docker data from the cont-1 container rootfs to a dedicated volume in the default storage pool that won’t be affected when the Incus server is updated from Ubuntu 22.04 to Ubuntu 24.04.
You could move the data back after the upgrade, but that’s not a requirment and perhaps this increases integrity for docker data in other ways.