My Incus server vmsmist refused to log me in via ssh or allow sudo commands three days prior to this video.
This is a tutorial on isolating the problem and finding a solution. If you have never experienced a major impact to services in your Home Lab, it’s likely that you just have not been running your applications long enough.
There are some key tips in this tutorial which can be of significant help to folks trying fix nearly unfixable scenarios.
My vmsmist Incus server hosts my core applications for my infrastructure.
As you can see, this server has 41 active and running containers. If you have not backed up each and every one of your Incus containers, go do that now. I had a perfect backup before this issue occurred.
I recall telling my Systems Admin staff more than once that I didn’t care if they ever did backups, however I did care if they could not restore all data to prevent loss. That means, not only backup your data, but test your backups.
A key point to having a successful server is to minimize complexity. For that reason, my Incus server does not run docker at the server level and my docker applications run inside incus containers. Likewise, my Incus server does not run other packages other than what is required to provide the incus containers access to resources.
The issue of not being able to login was precipitated by an error with one of my incus containers that should not have had any impact on the server.
The error was occurring with containers that had nested docker applications inside of them and so they had security.nesting=true on those containers.
The mqueue error did not immediately tell me a lot. The mqueue service is for inter-process communication and is a POSIX service that is a part of the Linux kernel. As it turned out, this was really just a symptom of the problem and not the cause.
As mentioned earlier, after I rebooted vmsmist I was unable to login via ssh and even when I did log into the console device, I could not execute a sudo command.
PTY’s are a software emulation of a physical terminal and they are used with ssh, sudo and the gnome-terminal as a few examples. The devpts mounted file system provides PTY devices.
My devpts mount had ptmxmode=000 which means no access and the correct value should be ptmxmode=666.
The mount point for devpts is:
The problem with what you see above is that this is supposed to have an associated symbolic link and it does not.
In modern Linux, devpts is mounted early in the boot process by systemd. Older Linux distros used to have an explicit mount for devpts in /etc/fstab. Here’s how the mount and the mount point should look.
After all my investigation, I am not sure I understand how this occurred. However, I went about booting from a live CD and chrooting into the damaged disk to try to make repairs. For those of you that don’t know, chroot (change root) is a very powerful Linux command that allows you to point to a Linux disk that is not currently booted (you do so from a booted live CD image) and you can edit internal files, install programs and even rewrite the initramfs responsible for early boot.
All this work nearly bordered on insanity and I started to question life itself.
After many attempts at chroot changes, I eventually just reinstalled Ubuntu server from scratch and restored my incus container backups (yes, I have backups). The problem is that I reinstalled from scratch five times and each time the problem persisted.
Note: Setting a Password on the root account might be your only salvation in problems like these where sudo might break.
A significant data point here is that I set my server up with Ubuntu Pro. Ubuntu Pro, also referred to as Expanded Security Maintenance (ESM), provides up to 10 years of real-time zero day security exploit patching. You get up to five free Ubuntu Pro licenses for your Home Lab. Since vmsmist is my most Internet facing Incus server (it runs NPM), I use Ubuntu Pro on that server.
Since I couldn’t solve the problem even with a scratch installation (more about that later), I developed a band-aid fix that is a systemd service that runs after incus starts during the boot-up process and it remounts devpts correctly and restores the symbolic link to the mount point.
I had been using nala to update my system because it provides more detail than apt.
sudo nala update && sudo nala upgrade
My upgrades were running great and I could clearly see that I was installing the latest Incus from the Zabbly repository as described in my Incus Containers Step by Step tutorial. Completely by accident, I discovered that only on vmsmist I was running Incus 6.0 LTS which was released in April 2024.
That was odd because I could see that Incus 6.15 was clearly being installed which is current as of August 2025. So, check which version you have actually running.
incus --version
My other eight incus servers were running the latest version, but somehow vmsmist was installing v6.15, but running v6.0 LTS. Here’s what I wanted, it’s not what I got.
In my Ubuntu Custom Desktop - Step by Step tutorial, one of the things I do is show you how you can override the Firefox Snap in Ubuntu 24.04 and replace it with the apt package from the Mozilla Firefox distribution.
It appears that by using Ubuntu Pro (ESM) that my vmsmist server had a higher package priority set for the incus coming from Canonical. I was installing the Zabbly version but transparent to me, the Canonical Incus 6.0 LTS was being installed as well. Even worse, the Canoncal version was set as the actual package that was running.
Pro tip: Here’s a command to look at any package to determine which versions are loaded and which one is actually running. I wish I had snapped a screenshot before I fixed it. Below you will see that both versions are loaded, but now the Zabbly version is the one that is correctly the default.
To make the Zabbly incus the one that is actually running, we create a policy file.
sudo nano /etc/apt/preferences.d/incus
Insert the following into the file.
Package: incus incus-client incus-common
Pin: origin pkgs.zabbly.com
Pin-Priority: 1001
The 1001 makes the Zabbly repository have a higher priority than the Incus version. Save the file with a CTRL O and enter and then CTRL X to exit the nano editor.
Perform an update:
sudo apt update
Make the latest incus from zabbly the version you are using.
sudo apt install incus
To verify afterwards.
incus --version
The second part of this is how to fix the devpts issue, because not being able to ssh or use sudo is a show stopper. Start by creating a script to fix the devpts mount and to restore the missing symbolic link.
sudo nano /usr/local/sbin/fix-ptmx.sh
Insert the following into the file.
#!/bin/bash
# Fix devpts mount and /dev/ptmx symlink
mount -o remount,ptmxmode=666 /dev/pts
ln -sf /dev/pts/ptmx /dev/ptmx
Do a CTRL O and enter to write the file out and a CTRL X to exit the nano editor.
Add execute permission to the file.
chmod +x /usr/local/sbin/fix-ptmx.sh
After much research, I discovered that this needed to be executed only after incus was completely started and apparently openssh-server as well. So, I created a systemd service.
sudo nano /etc/systemd/system/fix-ptmx.service
Insert the following into the file.
[Unit]
Description=Fix /dev/ptmx symlink and ptmxmode after containers and SSH are up
After=multi-user.target lxd.service incus.service ssh.service
Wants=lxd.service incus.service ssh.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/fix-ptmx.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Do a CTRL O and enter to write the file out and a CTRL X to exit the nano editor.
Any time that you create or modify a systemd service you need to reload the service directory.
sudo systemctl daemon-reload
Enable the service to run at boot time.
sudo systemctl enable fix-ptmx.service
To start the service now:
sudo systemctl start fix-ptmx.service
With the service in place, when I rebooted vmsmist, devpts was correctly remounted and the symbolic link was restored at the proper time to provide a functional system.
My summary thoughts are that perhaps if when reloading the server from scratch I had initially changed the priority of the Zabbly packages to install their version before I ever installed Incus, I probably would not have seen this issue at all.
If I had never run Ubuntu Pro (ESM) I might never have seen the problem. Even after changing to the Zabbly package as the preferred policy, I still needed to run my fix-ptmx service to have a functional system.
In any case, it’s probably a good idea to create policies for any repositories that you add to your systems. That way you can be sure that not only are you getting the latest updates, but that you are running the versions of the packages that you intend.
Hopefully this process will aid in debugging problems that you may encounter in your Home Lab.