These past couple of weeks I’ve been going deep into my Linux setup, way deeper than I probably should have for someone still learning the ropes. I’m running CachyOS with KDE Plasma on Wayland, and let’s just say it’s been a rollercoaster.
Disaster #1 — I thought I killed two drives
I was setting up my NVMe drives in fstab and mounted them incorrectly. The system just… stopped seeing them. A 2TB and a 500GB drive, gone. I genuinely thought I had bricked them both. Full panic mode. Turns out the drives were completely fine the whole time, just bad fstab entries. Fixed it, everything came back. Felt like an idiot but also learned more about fstab in 30 minutes than I had in months. (and along the way I got personally introduced to Linux emergency mode)
Disaster #2 — I broke my terminal chasing a pretty prompt
I ran the Powerlevel10k configuration wizard to get some nice icons in my terminal. Somewhere in the process my custom fastfetch setup (with a PNG logo) stopped showing up. Spent way too long debugging it, changed configs, reinstalled fastfetch, tried sixel, tried session files, nearly nuked everything. The fix? A single “type”: “kitty” that had been accidentally changed to “type”: “sixel” at some point during all the chaos.
Painful? Yes. Worth it? Absolutely. I’ve learned more about how Linux actually works by breaking things than I ever would have just reading docs.
Honestly, I know worse things are probably coming and at this point I’m okay with that. It’s the price of being curious, and I’ll take it <3
So… What’s your most panic-inducing Linux mistake? Bonus points if the fix was embarrassingly simple
Long time ago, my most stunning experience was when I completely borked my OS beyond salvation so I stripped and deleted almost everything including the desktop (using sudo apt purge && sudo apt autoremove extensively and liberally ) until I was only left with my $HOME, packetmanager, network and an emergency rescue entry.
Then a sudo apt-get install ubuntu-mate-desktop restored everything to working order.
I was baffeled how easy that was solved. I expected much more gruntwork.
Being able to extract knowledge and experience from real-life incidents is an excellent trait!
My two stories follow.
It was Ubuntu server with Zimbra e-mail server. Surely, Zimbra was not installed from Ubuntu repositories but from developer’s repo. And Zimbra did not use apt. It was time to update it. I tried that in test environment and everything updated fine. Oh, I had to configure and activate system-wide proxy to make Zimbra repo available. Update went smoothly. Nevertheless, test e-mail did not pass through server. Mind you, local e-mail. Well, that was early morning already and about ten thousands of users were at stake. Fortunately, I managed to fix things before the beginning of their working day. The reason was simple: I forgot to remove system-wide proxy setting after upgrade.
It was Red Hat server hosting client web-app. It was not my responsibility but I belonged to the second (or third?) line of support in corporate parlance or to the ultimate rescue team to put it differently. Considering its importance, the server had one-to-one backup residing on identical iron. One day the web-app at the main server was patched. All of sudden the server and the app admin found out that client sessions irregularly were lost and sometimes flowed as if the app was not patched at all. The mystery became even more puzzling when he discovered that one SSH connection brought him to the server and he saw all the patches in place and the next time he could not find a mere trace of them.
Well, did you know that Linux did not care if there was another server having that same IP address in the network? Lazy admin backed up the main server but neglected to turn it off afterwards! That is he patched the main server and the backup server went out of sync. As client as SSH connections went to different servers depending on pure chances.
@Zulan I’m impressed with your ability to articulate each of these “disasters” and explain them in an interesting article.
I multi-boot Linux distributions; my MAX on one piece of hardware was over a decade ago when I’d have between 12-14 different setups, including one real-time configuration - VxWorks; I may have included a Windows XP for the “challenge”, and the rest GNU/Linux varieties.
Somewhere in the past decade I was about to install something when I realize I was actually on a DIFFERENT computer than the one I thought.
I was installing to the entire disk (for the “other” hardware) when I should have been rewriting ONE PARTITION!
No problem-o! I have BACK UP USB Flash Drives galore! This means it was an OPPORTUNITY to reconfigure the disk and put the distributions I wanted to keep in a different order with an arguably better organization.
Here’s another one that was DELIBERATE:
When I was leaving a UNIX organization, we typically rewrite the workstation disks so that a different employee or new employee can use them - PLUS we had BOTH stable release distributions and Nightly Builds - so here’s what I did: sudo rm -fre / - our UNIX had the e option, so I could see this one SELF DESTRUCT until no I/O features were available. That was cool - and like I said, no harm was done because it would have been replaced anyway.
I wouldn’t quite call mine a panic, but it was building to that.
I have a windows VM on my work laptop. Fedora Sway is the OS with btrfs. I use virt manager. The first problem showed up when my VM would keep freezing. I started looking closely at it, and realized it kept pausing. That was a strange. I restarted the VM and it kept doing the same thing, just pausing. I thought, oh well another thing to fix.
So I jumped on WARP terminal, and told it to analyze. It did, used up all my credits, but it came back with a few suggestions, including check your disk space. And I am like, okay, but I have tons of disk space, but I checked it. I was down to 16 mb. Doh!!!
Hence the panic began. Where did it go? Who was taking up all the space, this is ridiculous. I started hunting for large files in my home directory. What did I download? I could not find anything. I calmed down and began to search for the largest files. But these were here long ago, oh well just to function, I began to delete some. The system felt a little freer so I deepened the search and then found it.
I had done a backup of my home directory which included the last backup into the actual home directory. A 150 something GB file. Doh!!!
Honestly, it was a lot for me, my whole life on Windows, and the smallest mistake feels like a real danger… Maybe I’m being too dramatic haha, but hey, I learned something!
Yes sir. That entire drive kill was done on a Windows 95 machine I think it was. I was at c:\ and did the del *.* /s … I was a teenager on my father’s WORK computer.
