My most painful Linux experience
Yesterday presented the biggest pain over my few years of using Linux that I have personally encountered. With the current prevalence of topics related to Linux, and especially ones related to new...
Yesterday presented the biggest pain over my few years of using Linux that I have personally encountered. With the current prevalence of topics related to Linux, and especially ones related to new users, I figured it would be good to share and leave a place for others to share any similar stories (and ideally how to avoid them).
The problem I encountered was effectively that my machine crashed and I was locked out on reboot. I'll describe how I crashed it later and for now just focus on the locked out bit.
During startup something was failing and as a result it would dump me into emergency mode. Emergency mode is basically just a root terminal where your ultimate goal is usually to read your logs and fix whatever was logged as failing. Annoying, but not a real issue. The real issue was that I was also locked out of emergency mode! This meant that literally the only thing I could do was get into a boot cycle telling me everything is locked.
So I head off to forums on my phone looking for what cryptic wizardry I'm going to need to perform. I need to a live boot OS because it is impossible to fix from my current install. I have to live boot another image, mount my original primary partition (after decrypting it), chroot the new mount point, and then use passwd to set a new root password. If I'm smart I'll come back to this thread later, when I'm not on my phone, and edit in or reply the actual commands needed since in reality I found myself piecing them together from across the Internet and maybe I'll need them again some day.
For avoiding this: check you have a root password. You may think you have one but might not. Set it to anything. Do it now, not after you're already locked out. The reason for being locked out of emergency mode was that passwordless root is locked, but there's no way to unlock it in emergency mode. I personally encountered this on Arch, but my search for error text was also taking me to Fedora forums so I don't think it is related to distro beyond the distro supporting no root password.
The bit down here is a bit less relevant as it is specific to my case.
Ultimately, I had an invalid /etc/fstab entry for a secondary drive (NTFS extra storage, not boot-critical). The thing is that entry has been there through months of daily boots and had worked, even though it may have been giving warnings or something. It's still lost on me as to why that suddenly became a boot blocker.
I'm pretty sure the original crash was my fault, although it seems pretty insane that what I was doing can break everything to the level it does. I was working on some Vulkan code and I definitely had some bugs in it that made my shader capable of reading out-of-bounds memory, but one would think this would stop at crashing the application. Instead it was causing graphical issues across the entire machine as if I'd simultaneously broken the logical drivers for every application, desktop environment included, at once. If I was lucky Plasma would reboot the whole desktop, if I was unlucky everything was completely frozen with mouse and keyboard doing nothing at all. It was me using the power button to escape the locked machine that triggered my chain of events.
For whatever reason on reboot it behaved differently than before. I'm still not sure why. I hadn't applied any updates or anything during that boot cycle. I shut this particular machine down every night and the issue was on a reboot, not my first boot of the day.