My experience switching to Linux and the need for guidance
Hello everyone,
This will be a long post because I want to give my post the proper context. I apologize in advance for taking your time.
About five months ago, with the help of relatively high ceiling of Windows 11's system requirements, I finally pushed myself to use Linux exclusively on my desktop. It was a decision between using Windows LTSC or Linux and I went with the better long term option.
I am not a programmer but I'm also not unfamiliar with the Linux world. I believe I've used one distro or another on a spare computer for shorts period of time since at least 2008. But those use cases have always been to satisfy the curious side of my brain as I am always interested in technology. So after installing distros ranging from Ubuntu to Arch, my curiosity waned enough to never look deeper into how these systems work. They were, after all, a hobby project on a spare computer that was often gathering dust.
When I decided to switch exclusively to Linux, the next decision I had to make was to pick a distro. Naturally, I looked for the established players first. Ubuntu was the obvious choice because it has long been the distro for newbies and there are a lot of guides on the internet if I ever needed help, which was inevitable. But then I read about snaps and thought that was a deal breaker. I was moving to Linux specifically because I don't want things shoved down my throat. I had no intention to relive that1.
So Ubuntu was a no go, but I was certain I wanted a Debian based distro as their support and software availability was unmatched, maybe save for Arch2. At this point, why not Debian right? It's known for being rock solid and it's Debian itself, not some derivation. Well, because I had various issues with Debian before. These issues were always fundamental and not very specific too, so I didn't want to risk wasting a lot of time fixing things I didn't understand, only for them to break again after a couple of days. Then I came across Pop!_OS, which seemed like a perfect fit. It was Ubuntu without its worst parts, came with Nvidia drivers and it had a company behind it that seemed to be committed to Linux. I installed it and everything just worked. I had zero issues.
But then I started getting that FOMO itch again. GNOME 42 was out and it looked great, but Pop!_OS was two versions behind. I also found out that they're working on their own DE, which might end up being great (it looked nice) but I didn't want to leave an established player like GNOME behind, including all the benefits you get from its wonderful extensions. I started looking for other distos again and Fedora caught my eye. I was obviously aware of Fedora, I even used it once back when YUM was still a thing, but it didn't leave a lasting impression on me. The fact that it wasn't a Debian based distro was also a disadvantage because that meant something different and at this stage of dipping my toes into Linux, I didn't think different might be the best way to go for me. Still, despite my best judgment, I installed Fedora on a USB and used it live. When my gut feeling was confirmed by my research about how Fedora leaves things as stock as possible and is ahead of the curve in terms of upcoming technology (btrfs, PulseAudio, Wayland et al.3) without sacrificing on stability, I was hooked.
After renewing my Timeshift backup, I formatted my Pop!_OS system and installed Fedora. The installation process could use a facelift, but it handled everything perfectly. I didn't even have some of the issues I had with Pop!_OS right after installation. It was literally problem free. I'm now on day #3 of using Fedora and the experience remains the same. The only issue I had to deal with was trying to get Timeshift to work (apparently it doesn't play nice with btrfs on Fedora), but instead of wasting my time with that, I just installed Déjà Dup and I'm good to go again. Barring any drastic issues, I don't plan on changing my distro again.
Now, onto my plea for guidance.
I'm looking for comprehensive resources that will teach me how Linux works under the hood. Considering my non-programming background, I'd appreciate it if the language is approachable. The reason why I want this, for one thing, is to learn more about the system I'm planning to use probably for the rest of my life (in tandem with macOS) but also, I want to do some cool stuff Linux allows users to do.
Just to give a quick example. Yesterday, I installed Rofi, which is, besides many other things, an app launcher. I got it to work just fine, I even got a configuration of my own with a theme of my choosing, but when it comes to using some scripts, I just couldn't do it. Every video I watched on YouTube told me how easy it is to use scripts with it as if it's a self-explanatory thing, but I was simply clueless. There was a lot of lingo thrown around like environment variables, setting up $PATH
, making the scripts executable with chmod
etc. I have very little knowledge of these things. I want to learn what they are, why they exist, and how they all tie together. I want to learn how /etc/
is different than /usr/
and the difference between X11 and some DE (or if they're even in the same category of things). Now, at the risk of sounding impatient and maybe even worse, I also don't want to go way too deep into these things. I am not, after all, trying to become a kernel developer. I just want to be better informed.
There are a lot of information on the internet but most of this information is scattered and out of context. If I try to learn more about one thing, I'm bombarded about other things that I don't know, so in the end I learn nothing. In short, I'm looking for a comprehensive, entry level video series or a book about Linux written in an easy to understand language that assumes no prior knowledge.
Additionally, I'd appreciate any website, YouTube channel and what have you to keep up with recent developments in Linux. I already found a couple as there are plenty of them, but I'd like to learn more about how people here keep up with this fast changing environment.
Thank you for reading and sorry for being so verbose! 😊
1: I know you can remove snaps, but I didn't want to deal with the hassle of any possible issues deleting a core system functionally might bring about.
2: Despite finding its approach fascinating, I had no intention to get into Arch because it's a rolling distro and I didn't want an advanced system that can break at any moment in the hands of a novice like myself.
3: To be clear, I don't know how most of these technologies are better than alternatives, but the Linux community at large seems to think they're drastically better than alternatives and are the future.
Yay, another fedora user!
For linux resources, its a tough task. As you note, there is a lot of information and depth there - and the level of "common knowledge" assumed is often different for many guides/walkthroughs/instructions. The good news is that a lot of the basics haven't changed too much. So if you find an "old" guide, it may still be perfectly relevant - for example, anything covering basic shell stuff. I don't necessarily have a perfect path of materials, but a random dump of stuff:
.bashrc
- one of many guides that covers the bash preferences. Basically a shell script that is whenever you open a new terminal window. If you spend much time in the command line, you'll tweak this over time to your preferences. Feel free to ask other devs for their cool bashrc hacks (I'll add a few in a reply to this)$PATH
variable - since you specifically mentioned it =)/etc/
,/usr/
, and all the other top-level directories.Hopefully some of that is helpful - many guides have similar options written by other people, so if one particular guide doesn't click - another option is usually just a search away.
Alrighty, some random things you can do with
.bashrc
. You'll see the alias command a lot here, which is probably self-explanatory...but basically after the alias is set, if you type the string on the left, it will actually do the thing on the right.Side note, it might be a fun thread to share their bash profiles, so we can all see some cool things and build our own up =)
I grant ye the power of The Fuck
While, since I'm a relatively new Linux system admin, I'm still learning- thank you for these links- they will be helpful even in that capacity. My day job doesn't necessarily require that I understand how linux works under the hood- but that understanding will help me perform what I do more efficiently, regardless.
I am about to make the jump myself on my main desktop PC at home, too.
Already did on my personal laptop- and I can strongly second the Gnome Tweak Tool and Extensions, even for distros that do things their own way (Zorin), though you do have to be careful of settings in two areas conflicting with one another. Having access to easily change so many extra settings made my experience so much better in general, though. I'll never go back to Windows on my laptop. I've converted there for good and hope to do the same on my desktop!
Hey, distro friend!
Thank you for your reply, that's a lot of resources.
The Linux Documentation Project sounds exactly like what I'm looking for. I will definitely be reading that and taking notes. According to its Introduction section, it looks like it's aimed at people like me.
I'll be adding the Fedora specific resources to my RSS reader.
As for bash, I'm actually using zsh. If my understanding is correct, it's pretty close to bash and most bash scripts work perfectly fine with it but that's what I'm using. If you were to ask me why I use zsh and not bash, I wouldn't give you a satisfying answer other than I saw this thing, it looked pretty, and I wanted to have it. 😄
Linux Directory Structure page looks awesome too, thank you. Exactly what I wanted.
I've familiarized myself with Gnome extensions. Right now I have Always Indicator, AppIndicator, Blur my Shell, Dash to Panel, Hue Lights, Just Perfection, Media Controls, and Volume Scroller activated. All work perfectly fine. I did see caffeine before but I never let my computers go to sleep, so I just changed the power settings instead.
Your comment was indeed helpful, thank you again.
Hey so if anyone wants a zshrc to start from, I made this one a long time ago and have been tweaking it ever since:
https://github.com/jleclanche/dotfiles
I know a lot of people like it -- it's gotten 150+ stars and at some point it even became part of the standard Spotify linux setup.
cc @skullkid2424
O.o
I might not zsh, but I'll definitely be stealing a few of those aliases/functions and adapting them for my bashrc.
That looks pretty good! I'll bookmark it. My Terminal currently looks like this. It's relatively stock but I like it so far. Yours look definitely more compact which I might have to switch to, but right now I have very minimal needs from my terminal so I mainly focused on how it looks, rather than how it functions.
Happy to help =)
Using zsh should be fine - there are a few differences that can come up, but 90% of the time anything that works in bash should work the same in zsh. It is probably overkill for non-devs, but thats fine. You can customize the prompts for bash as well, depending on the level of detail you want. Looks like that tool goes pretty in-depth.
I myself use the following in my bashrc, which changes my prompt to my preferred color schemes and includes the active git branch. Basically certain strings change the color the text (which is what you see with the
BLUE="blahblahblah"
lines). And then the main prompt is PS1 - so you basically say the color you want, and then the rest of the prompt. My prompt basically has an opening bracket in light blue, hostname in blue, current working directory in green, git branch in red (if there is one), and then resets the colors. Not to discourage you from zsh - just sharing more fun stuff.It's definitely an overkill for me. I mainly use the Terminal to for ncspot and ticker so I don't even get to see the aesthetics but it looks nice when I do.
I applied your settings to my
.bashrc
and it looks pretty nice! I think I'll keep it with a little bit of tweak. I'm sure I'll have to switch to bash once in awhile, there's no reason why it doesn't look nice as well. Thanks! 😊zsh is pretty neat, but you'll never have to worry about your scripts' bash compatibility if you use a bash shebang
#!/usr/bin/env bash
at the top of the file. That will instruct your computer to use the bash interpreter (or python/perl etc.) instead of your current shell default.Ah, that's good to know! I guess it's like
<!DOCTYPE html>
? That makes perfect sense. Thank you for the info.I don't, particularly, recall reading anything much about how linux works as a whole. I just did a lot of various things over many years - from installing a printer to setting up a DNS server or making a script which reminds me to put the bins out and everything inbetween - and read about how to do that specific thing, all of which built up my background knowledge.
When job X requires prior knowledge Y, I'd go off and read about Y until I had enough understanding to get X done. So if something was assuming I knew how to use $PATH then I'd go off and read a bit about $PATH - which would then lead me to reading about environment variables, then make my way back to the original topic. Sometime I just copy and paste stuff from Stack Exchange or Debian's forums until it works or I understand it (or both).
You could probably do a lot worse than just spending a few hours on wikipedia, starting here, frankly. It's pretty well written, detailed enough and anything unclear you can just click on to find out more.
Then, when you want to do the "cool stuff" that linux lets you do, just do the thing. Task-led learning is a good way to learn stuff. Sometimes it'll take a while (days, maybe more!) and require a lot of background reading, but eventually you'll need to do less and less of that.
Thank you for your reply.
Spending time with it will certainly help. I already have a folder in my note taking application documenting every issue I've encountered and what to do about them.
As I implied in my post, I don't want to go from resource to resource, trying to learn one thing after another in an unstructured manner. Unfortunately, my brain doesn't work that way. That's why I wanted a contained book or a video series where things are explained with little or no prior knowledge, gradually going from subject to subject to connect everything together. Maybe I'm asking for the impossible because Linux isn't built that way, I have no idea. I'm sorry if my request sounds dumb to someone knowledgeable.
Practice can certainly be a good way to learn and I will inevitably know more than I do as I use the system but sometimes you hit a road block where things just don't make sense to you. That's what got me thinking that I need a more structural approach to learning, rather than breaking things and mindlessly copying Terminal commands from Stack Exchange with no intention/involvement behind it.
You may be looking for How Linux Works: What Every Superuser Should Know. It is very long, and it may be more detail than you want, but I have yet to find any single source that is more comprehensive (except maybe Wikipedia, but quality varies between articles). It is also quite expensive, but you may be able to find it used, in a library, or elsewhere for less.
(Disclaimer: I only have the 2nd Edition.)
It starts by introducing the three layers of the hardware, the kernel, and userspace, then going over each in basic detail, and then introducing how the users (and root) work in userspace. The second chapter is the absolute basics of what a shell is and running commands (including an explanation of
$PATH
). It then dives deep into more fundamentals, before first introducing scripting in chapter 11. With all the base knowledge established, it speeds up, introducing compilation from source in chapter 16, and apparently the 3rd edition even goes into virtualization after that.That book indeed looks comprehensive. I'll definitely bookmark that after checking up on @skullkid2424's suggestion (PDF). I doubt I'd find it in a library anywhere around me since I'm not in an English speaking country known to have libraries with selections but I can certainly buy the e-book version if TLDP's structure doesn't agree with my brain. Thank you for a very detailed recommendation, I really appreciate it!
I just wanted to say thank you for letting me know about this book! I've been using Linux sporadically since 2006 or so and I know the broad strokes of how to use it but never really learned how it works.
I'm about halfway through the book and I've already learned more from this book than I have any other Linux resource.
Pieces of the ecosystem come and go, but the fundamentals haven't changed much since the 90s. It sounds to me like you want a basic understanding of those fundamentals so that you can interpret (and use!) the new, flashy stuff.
There is one book I can recommend that will give you those basics in one place: O'Reilly's Learning the Bash Shell. It was first published in '95, and hasn't changed much. It's not even specific to Linux. The content is much more about interacting with software than programming. You don't have to read it cover-to-cover, just aim to soak up the big picture from the first few chapters.
Your PATH, the file system, and those magic command line incantations that don't make any sense right now will become much clearer.
Thank you for your comment!
This is exactly what I want. The flashy new stuff that I mentioned (in your second quote) was meant for stuff like recently released applications and whatnot.
As I mentioned to @skullkid2424 in their very helpful reply, I am not using bash but I think I'll still need to read up on it since it looks to be an important component of the system. So I'll check that book out!
First: Welcome to the club. Linux can be daunting, but don't give up too easily.
The problem here (and it's not a big problem, so don't let this dissuade you and go back to Windows) is that there is some variance in Linux land. OpenRC vs. systemd. Gnome vs KDE (vs other alternatives). The plethora of package managers and package types (apt, deb, yum, pacman, portage, tar.gz, and more). Shells (bash, zsh, etc.) So, you kind of want to keep your scope or boundaries in check. Know what your chosen distro and local system are using or have chosen among the alternatives, so you only spend time learning about those things. It's a waste of time to read about OpenRC stuff when your Linux is systemd.
Probably you can start getting familiar with this kind of stuff just on an as-needed basis, so you don't overwhelm yourself. To give you a springboard to skim the Internet and learn more elsewhere:
DEBUG
istrue
.PATH
is an environment variable, and a very common one. It is used by the shell you're using (bash, zsh, or whatever). It contains a concatenated list of directory (folder) paths. An example of aPATH
value might be/usr/bin:/usr/local/bin
, which has two elements, separated/joined by a colon delimiter (the conventional delimiter inPATH
). When you try to run an app by name in your shell (such asls
), your shell looks for that app in the dirs of thePATH
. Ifls
is in one of those dirs, thenls
will run. If not, it won't, and your shell will return with some error message.chmod
is short for "change mode". Normally, created files are not executable. So, if you have a file calledfoo
in your current dir, and you try to run./foo
in your shell, it won't execute. We have to make the file executable first, and the usual way to do that is with chmod. In particular:chmod +x ./foo
. After that, it can be executed with./foo
. You might wonder: "Well, that's odd. If I have something that's meant to execute, why doesn't it just get created as executable?" Well, scripts are just files with plain text in them, and not all text files are meant to be executed (some are just meant to have their contents read). So, we need to tell the system that a given text file contains script code, and is meant to be executable./etc
traditionally holds system-wide configuration, and is traditionally only changeable by theroot
user/usr
usually holds ... well, a bunch of stuff, and what kinds of things often depends on the distro. The files of most installed apps will live somewhere under/usr
. A lot can be said about/usr
, but I'll keep this message brief.Hope that helps. Try to find places where you can ask questions, such as the forums or chat channels (IRC, Discord, whatever) of your distro of choice. In my opinion, you'll learn better (retain better) and faster by asking pointed questions about specific issues you're having, or specific questions you have, rather than trying to learn from an organized course or book.
Thank you for your encouraging words and detailed reply.
Thankfully, I have enough background to differentiate between the desktop environments, package managers and most of the other stuff Linux environments presents to users. The information I lack is what you so kindly provided in the other part of your comment where you go through the directory structures and other stuff.
Specifically, the way you explained
chmod
is perfect. You explained to me what it is but not only that, you explained why it doesn't work the way any normal person would assume it'd work coming from more consumer friendly operating systems like macOS or Windows. That's exactly the kind of information I'm looking for. You should write a book! 😀I'll try to heed your advice on joining communities. This I'll have to do carefully though because Linux community can be, well, we all know how they can be. Gotta make find the right ones to not sour on the whole thing.
Thanks again for your comment!
I'll hijack @pistos thread to say, I once believed I needed to understand the origins and reasons for why the Linux base file system was the way it is. I thought understanding the logic for /etc and /var would help make it simpler to use. What I discovered with use and time was that often the answer is unsatisfying (from a usability stand point at least) and it often comes down to "because that's where the first person who did it put it".
There are still gaps in my knowledge today - I've never piped /dev/{device} anywhere - but those are things I saw power users doing 20 years ago because Linux was not as user friendly as it is today, and most of the non-obvious stuff I need to do for my job is well-trodden territory that has lots of resources online.
So don't feel like you have to understand BEFORE you just start doing. A lot of it you'll just pick up by osmosis over time, and seeing the same things come up as you fix problems and try new things. It sounds like you have the basics of navigation, and that's often enough to dive in.
Hey, thank you for the advice. I certainly don't aim to just learn things just by the book. As I mentioned I've been using Linux for months now and doing day-to-day stuff has taught me a lot. I assume it will continue to do so.
The books will be helpful in so far as they will be my guide whenever I'm stuck and because they have inherent structure, I can expand my knowledge if I wanted to in an organized way. I certainly don't plan on studying Linux subsystems few hours a day just by reading a book. I can't do that even if I wanted to because there are only 24 hours a day and I have responsibilities. I'll read them at my own pace and I'm sure I'll learn things I'd never thought to ask, but my main plan is to use my system (or another device if I'm feeling adventurous) like I normally would, gradually improving my understanding day by day. Dipping my toes into the ocean one day at a time to see how it feels, as it were. But regardless of how it feels, I'm not going to jump into it simply because this is just a hobby, an itch to scratch.
Nothing that you wish to learn and do is distro specific so stop changing distros. These things take time, I don't think there's any deeper advice that I can give you. Just use the Arch wiki and make specific questions on /r/linuxquestions and /r/linux4noobs. Read some manpages. Rofi is really not that hard, you just gotta be used to config files. They're often at
~/.config
. Read the book The Linux Command Line to make your life easier.I indeed meant PipeWire. Chalk it up to my ignorance. Thank you for correcting me and for the links. I wasn't aware of Phoronix, it looks great.
Linux is easy, just RTFM!
(I did mean that as a joke, but on a more serious note one of the things that will help you the most is to know how to use man and info to find the information you need.)
There are many different layers of mastery when it comes to computer systems, so to answer your question we would need a clearer definition of what it is that you want to learn. IMHO if you posted this thread from a computer running Linux then that means you already know how to use it! Most people don’t need more knowledge than that.
I will also parrot @lou here: distribute hopping isn’t helpful. If there is a program or feature you don’t like there are ways to disable or remove them, and if something is out of date there are increasingly easier alternatives to get updated versions. But as you switch you might notice that the software can be set up dramatically different and some things may not work with the security model they have implemented.
One source that helped me a lot when I was learning system administration were trade paperbacks. They are typically comprehensive at the expense of being out of date, though having some things being out of date won’t be terribly bad because systems designers generally care about stability and old features typically get gradually phased out.
Unfortunately it’s been far too long since I have needed to read any of them so I don’t have any particular book to recommend. Just make sure you get something that was published in the last year or two, or better yet find something in preproduction! It’s not uncommon to see preproduction tech books available for public comment in the FOSS world.
One last specific piece of advice: learn the ins and outs of vi. You don’t have to make it your editor of choice but you should be familiar with how it works because a number of legacy tools tend to use the same controls. Beyond that, it’s also the most commonly installed editor on Unix systems, so it’s a useful tool to have.
Thank you for your comment.
I certainly aim to read the manuals. The issue is some of them have those pesky lingo that pass by me and they assume prior knowledge. English is also not my native language, which is often not a problem but I didn't have a technical education, nor any opportunity to truly deep dive into technical reading, so some technical terminology can make the text harder to grasp for my old brain.
I hesitate to bring this up because it will sound like I'm accusing you, which I most certainly am not, but most people in the Linux community seem to think every person can find the answer they're looking for in the manuals. I'm sure that's true for anyone who have a basic understanding of what they're trying to accomplish, but if you're a novice like I am, that advice is a bit akin to this. I am a convert, so I don't need to be convinced of the benefits of using a Linux system, but people should stop treating it like everything is oh so easy. Sorry again for the mini rant that has nothing to do with your comment!
I may have misspoke in my original post, considering yourself and @lou advise me to stop distro hopping. I am certainly not doing that and I'm fully aware that changing my distros is not going to help me learn about fundamentals of Linux. I was doing distro hopping, but that was when I had time to kill with a spare machine. Now I'm using Linux on my primary desktop computer, so I'm not going to screw around with a new distro every 3 months and risk my data and my settings, despite already having two separate backups.
Thanks for the vi advice. I didn't know that. I've always use nano for easy editing and VSCodium for larger stuff like config files, but I'll try to familiarize myself with vi.
Just remember that you don't need to read the manuals like a novel. You may instead learn how to quickly find whatever you need in them, and use them very often.
There is almost always a good source of information for any difficult Linux system, but that is very unlikely to be its manpages. Those are supposed to be short reference pages that you can quickly search through to find what you need so you don’t have to memorize special switches that you only need in obscure situations. They are a good first source to look for answers because they are dense with information but they aren’t going to have everything in them by nature.
So far in my experience, I agree with you. I also find them to hard to navigate but that's obviously not its fault. I'll just need to get used to it a bit more. The thing is, it's not just about finding the right tool either because my brain has not been wired for any of it after 20+ years of using computers in a completely different way with Windows and macOS. It will take time but hopefully I can adjust to the way Linux does things.
Oh boy. I'd start with a quick primer on the linux file system. There are only a few parts of it you'll care to interact with regularly, it's not as scary as it looks. Then you can dive in to this book on the command line. It's free, current, and aimed at new users. You might find linux books as good as this one, but you won't find a better one.
Learn the joy of
man
andapropos
so you don't go crazy trying to remember everything.A few people pointed me to different resources on Linux's file structure so it seems like a subject I need to have a firm grasp on, which I wanted to do in the first place. Your resources look pretty good too, I especially like the hierarchy graph at the bottom of the page.
If I recall correctly, I might have the book you linked. It must have been in a Humble Bundle I bought a couple of years ago. I'm sure I'll be able to find it from one of my old backups.
I knew about
man
but had no idea aboutapropos
(what a great command name, too). Sounds like something I'll use a lot. Thanks!This is a pretty simple (and free) interactive course that I have pointed to co-op students and similar at in the past, it will teach you some of the very basics about bash and interacting with linux from a terminal.
Oh, that looks like an amazing resource! I'll spend some time on this tomorrow for sure. Thank you so much!
Excellent suggestions have been made about what to learn and how to accumulate knowledge of linux and its ecosystem. I'll make my points on what I consider the groundwork, leading up to Linux From Scratch, which may appear daunting but is in my opinion one of the best resource for wrapping your head around linux systems.
I advise you to also read OpenBSD documentation for each topic below, to learn more about system administration and *nix systems.
Filesystem
Understanding the filesystem is a great but deep well to drink from. Common *nix filesystem hierarchies is a nice place to start. A lot of inexplicable decisions stem from very old hardware limitations.
Shell
Familiarity with the shell is great and certainly a productivity benefit, but I suggest you start with posix shell to avoid picking up bad habits. Also, newcomers are sometimes advised to come back to the shell later as it can be a little frustrating. Tools like shellcheck and the pure sh bible might help you learn safely. Understanding the environment around the shell is also helpful: the virtual terminal.
Common System Utilities
Pretty much every linux distribution shares some common ground utilities for convenience or necessity. Understanding which ones are core to the operation of your system is a good learning experience, but unlike OpenBSD, linux can be stripped to its bones much more easily. I advise you to learn about OpenBSD's base utilities, and then learn about linux's. Good resources for that are efforts at clean, minimal, and easily maintainable distributions. In particular Kiss Linux and Oasis Linux are wonderful examples. Also, the popular bundle for essential utilities, BusyBox, and the alternative Toybox.
System Init
I consider there to be three discrete parts to understanding linux systems in operation. Building the system, which is usually done for you by distribution maintainers, system bringup, performed when you power the computer on, and system maintenance, which is the majority of the time spent in operation and is frankly easy enough learn by experience.
How the computer arrives at the familiar linux environment is the business of system bringup, and is core to understanding all the required moving parts. Some of this domain is tied up in hardware specifics, firmware, and BIOS/UEFI (at least on x86). I do advise you learn about what exactly is involved in this process, resources include SeaBios, Grub, CoreBoot. However, the priority is the linux init system. The best way to learn is to study init systems and their designs. Skarnet.org's s6 is particularly instructive, but runit should be understandable as well. When you understand the requirements for an init system, an advanced exercise could be to replace an init system (on distributions that take care to make this relatively painless like Void, Kiss, and Alpine) with one of the extremely simple inits, like suckless' sinit, or even s6 if it strikes your fancy.
Linux From Scratch
At this point, there should be no administrative problem you can't figure out how to solve. You know the layout of the linux filesystem, and even some of the underlying mechanics. You can wield the shell or at least diagnose issues, and your toolbox contains the common system utilities. You might have even dipped your toes into make and compiled a software package or two from source.
Try Linux From Scratch, at your own pace. Ideally you have a clunky old computer for this project, and your primary machine remains production ready. This is really important because it will reduce any frustration or sense of urgency you may experience during the process. It might take an attempt or two, depending on what issues might arise and if they put you off the project for a while.
Ultimately, however, the documentation is good and the learning by doing is excellent.
A lot of this sounds pretty advanced for me at this stage, but I'll definitely keep some of your advice in mind. Especially the part about some of the Linux design decisions being made solely out of hardware limitations at the time.
I don't know how this learning process will go, but if I end up wanting to go deep, it looks like I'll need to source myself a spare machine for tinkering, like the old times, except this time for real.
Thank you for taking the time!
Happy to help! If you don't have a spare clunker lying around, I highly recommend obtaining an old beaten up Thinkpad, ideally an x230 or comparable, and a cheap sata ssd. Should be around $200 and has an ethernet port.
Alternatively, if you have an old monitor, keyboard, and space, or can requisition one when not working, maybe a cheap intel Nuc?
Best case, you scrounge online for a while and pick something up for free.
It just occurred to me that I have a Raspberry Pi lying around. I used to use it for Pi-Hole but I don't maintain it anymore so it's just gathering dust. It could be perfect for this type of thing. I don't want to get ahead of myself, I hardly have enough time because of work to begin with, but I'll make a mental note for this. Thanks again. 😊
Edit: Just noticed your additions to your original post. Thank you again, I really appreciate it. This thread has turned into a hopefully very useful starting point for anyone wanting to learn more Linux, which is pretty nice!
Either way, keep in mind that there are going to be some differences depending on what kind of firmware is installed on the machine. For ARM and some of the less common platforms you'll probably run into OpenFirmware. Though even with x86, depending on how old it is and how you have it configured, you'll be dealing with either EFI/UEFI or "legacy" BIOS booting.
Though IMHO I don't think it's terribly necessary in these days to be able to compile your own kernel at all. There's a lot more useful things to learn, so I would save this kind of thing for last. If you understand scripting, networking, security, and how to manage kernel modules, you're already a Linux Guru. Becoming a Linux God is not as useful as a title as you would expect (unless you plan on building a career on it, of course).
Yeah, I don't need to get "close to the metal", as it were. The ideal place I want to reach is what you described as Linux Guru. Even that might be overachieving for me but that'd be a great place to be in a year or two. I don't want to make a job out of this or change my career as a Linux sysadmin. I just want to get to a point where if I see an error message, I can figure out what the problem is simply by reading the output, and if I want to automate a task, I want to use
cron
to run the script I wrote for my tailored needs, and stuff like that. I've certainly gotten enough help to get there, now it's up to me. 😊Now with links :)