Can someone explain the systemd controversy to a nontechnical user?
A project I'm working on requires me to cover a bit of comedy targeting Lennart Poettering as it's tangentially related, and I'd like to have more context even though it's not strictly necessary. I'm a nontechnical Linux user who used the OS before systemd came around, but really the only impact on my life it's had is that I occasionally use systemctl to control services.
Though I wasn't paying as much attention to the community around the time major distributions switched, I've been casually exposed to criticism of it ever since I came back, and I'd like to make sense of it all and form an opinion beyond "I like Fedora and GNOME and it seems to go hand-in-hand with those". I've read The Biggest Myths, the Wikipedia article, some stuff on freedesktop.org, and of course absorbed the venom slung back and forth over systemd in every FOSS community, but it's hard to get a full picture. And a picture from 2022, for that matter, as a lot of this information comes from its early days. Help me out?
The simple version, because the rest turns into personal attacks on various people or projects:
The "Unix philosophy" is a starting point for it. "Do one thing and do it well." Systemd is a Swiss army knife, so it violates what many people see as a major tenet of what a good utility design philosophy is.
It was also, quite honestly, rolled out a bit before it should have been, which combined with bog-standard change aversion, created a monumental pushback.
sytemd violates an old Unix design philosophy regarding software utilities and components: "Do one thing and do it well". Before systemd, the popular init system (simply called init) was very simple in it's design. You essentially gave it a list of things that needed to be done to initialize your system, and it performed them in sequence.
The problem with this was that it resulted in very long boot times, because the initialization process is done synchronously (one task after another). systemd sought to solve this in two ways:
That last point is the biggest area of contention. systemd has absorbed many simple services people were very familiar with into one big complex monolith. In some ways, this makes things harder to tinker with. The parallel approach to initialization can also make troubleshooting boot issues more difficult.
Personally, I'm in camp systemd. The radical improvement to boot times it has brought us are great. I find using it to manage services (and building my own services for it) pretty intuitive after learning it. Most of the Linux world has pretty much moved on from this battle, and few major distros still offer init. But I understand where the dissenters are coming from.
Small note... unless I'm misreading your post, synchronous doesn't mean one after another, it means "at the same time". As in synced in time. Likewise, asynchronous doesn't mean parallel, it means "not at the same time", though the two can coincide.
The terms would instead be serial versus parallel.
I did some googling and you're right. Somehow I never encountered this usage, or didn't retain it when I did.
It was a mini rabbit hole as I'm not the first person to find it interesting that synchronous was borrowed for this purpose in spite of it's definition. Especially given that sync and async already existed in electronics engineering in the context of data transmission, both being strategies for serial communication.
Since this is probably interesting only to me, I'll skip the rest of the rabbit hole, @babypuncher was definitely using the terms correctly.
Not only interesting to you. When I was first learning async programming I found the nomenclature to be really weird and unintuitive for the same reason. I prefer the words "blocking" and "non-blocking" in most cases, as they make their implications a lot more obvious up front.
As someone who works in software with a background in hardware -- it's mind boggling the definition stuck. Words mean things people!
In the context of programming languages, it could be referring to the relationship between the code execution and the action, not between different actions. The synchronous code's action happens at the "same time" that line is executed, and it doesn't move to the next line until complete. Meanwhile, the asynchronous code's action happens sometime after that line is executed and processing has already moved on to later lines.
Here is a history I came across a while back. Most of the complaints I’ve heard are along the lines of “it’s too monolithic” while ignoring the fact that the kernel is a monolith.
This is a fallacy. Specifically,
Fallacy #1.2: "The Linux kernel is monolithic, therefore it is okay for systemd to be monolithic"
This is also a non sequitur. Whether or not systemd should be monolithic has absolutely nothing to do with the Linux kernel.
Quoted from https://judecnelson.blogspot.com/2014/09/systemd-biggest-fallacies.html
That's not really fallacious; at least, it needn't be, as such. Suppose there is a contingency X which says: ‘I like Linux, but I want an init daemon other than systemd, which is bad because it is a monolith’. The implicit premise is that monoliths are bad; i.e., the argument runs: ‘all monoliths are bad; systemd is a monolith; therefore, systemd is bad’. If contingency Y responds with: ‘The Linux kernel is monolithic, therefore it is okay for systemd to be monolithic’, really, what they are pointing out is the hypocrisy in X's argument.
Now, this may not actually be the argument used by any particular X, or by most Xs. Perhaps they instead say ‘it is bad for init daemons to be monolithic; but it is fine for kernels’. And this is really the problem with assessing such arguments in a vacuum; there is always context. It's true that ‘The Linux kernel is monolithic, therefore it is okay for systemd to be monolithic’ is a non sequitur, but I highly doubt anyone was simply saying that without further context. And the linked post does not actually cite anyone who makes such an argument, in which respect I can not see it as anything other than a strawman.
(FWIW this should not be taken as an opinion regarding systemd. I will not express any opinions regarding systemd without a lawyer present :)
EDIT: I skimmed, and completely missed that the parent did in fact make exactly the sort of argument I claimed was nonexistent. I must eat my words!
Yep, the parent made the argument. And while I appreciate what you wrote, it is indeed fallacious. The individual who has stated that "systemd is bad because it is monolithic" hasn't made any statements about the Linux kernel. The responder is bringing up something irrelevant to whether or not systemd is bad because it is monolithic.
Also, it isn't necessarily hypocrisy if someone believes it is bad for systemd to be monolithic and good for the Linux kernel to be monolithic because they are different pieces of software that do different things.
Is it not still a valid (if not directly spoken) question being asked? "Why is being monolithic here bad when it's not over there?" I've never seen an answer to that.
Generally a statement is being made, rather than a question being asked. But if you would like an answer...
I don't personally think kernels should be monolithic. I prefer microkernels from a design standpoint. From a practicality standpoint, the Linux kernel and the BSD kernels are what they are and work so I won't be switching to SeL4 any time soon. But even monolithic kernels have advanced with loadable kernel modules and compile time options to remove unnecessary features, which an astute observer might notice was not the case with systemd.
Edit: to restate as to why someone might think it's okay: the person stating that systemd is bad for being a microkernel isn't commenting on the Linux kernel. And even if they think it is okay for the Linux kernel to be monolithic, it might be because it is a different piece of software that does something different. Or perhaps because, as I wrote above, the Linux kernel handles being monolithic much more gracefully.
Many linux users were worried about systemd because it is a different kind of init system than usual, while linux community were used to be able to configure which init system their computer used. It seemed like - with red hats corporate influence - it would be more likely than other init systems to do an embrace-extend-extinguish type thing and become the only init system. This is essentially what has occurred. Distros today without systemd as the default are very niche.
I think a lot of people had trouble with pulseaudio and were worried systemd (by the same author) would bring along a similar amount of problems.
It's also useful to read about how systemd ended up in debian https://lwn.net/Articles/806332/
Seconding. And a big part of it as well was that systemd tried to unify a lot of system level services, both by bringing them into systemd itself, but also sometimes by confusingly bringing them under the systemd “umbrella” without them necessarily being part of systemd.
So when faced with this, people saw systemd as replacing not just init but also their logging systems, lots of config files, cron, network config, user management, and more.
Linux Weekly News reported the outcome of that vote here.
It's interesting to see how popular it is, because from seeing rants about it online it's easy to get the idea that it's much less popular than it really is.
Though is it just me or is the post the absolutely most confusing way to deliver poll results? Why does it not lead with a list of the vote counts? Why does every option have both a number and a letter? If I was trying to present poll results in a ridiculous way as a prank then I couldn't do half as good as this!
Part of it is that they’re not using regular voting. It’s some kind of ranked preferences system, but more complicated.
Lots of things that didn't need to, and didn't use to, developed a dependency on systemd or on software that became part of systemd's kitchen sink. So the web of dependencies grew quite large.
Iirc this was also around the time that Daniel J. Bernstein's init system, daemontools, was becoming trendy as a very well designed, small piece of software. There was controversy there too, because OpenBSD removed it after licensing and filesystem hierarchy differences could not be rectified.
So I know I had something cutting edge to compare systemd to, and certainly wasn't interested in systemd as a unifying init and rc and event bus and everything else system going forward.
For what it's worth, I only ever heard from package maintainers that they really appreciated only needing to write init scripts/unit files for systemd, but I can't say I appreciate that at all. And I'm sure I haven't heard the whole spectrum of thoughts.
Edit: you could check out skarnet.org's s6 (an init system with optional rc and other components), which I feel is the best documented case against systemd in principle and in practice.
A little off topic but I've been annoyed for a long time that djb's various tools didn't make it as defaults in the big distros. In the old days of email, qmail was peerless (not to mention Maildir is so much easier than mbox), djbdns is incredible, daemontools brilliant.
He's a little idiosyncratic but damn the guy can write fast, secure, absolutely rock solid stable software.
Due to certain criteria he put forth, I think it was destined to remain more of an example, similar to his vision for make, "redo", which he never even released. These days there are functioning implementations though.
Also, even daemontools was not flawless, and there have been improvements IMO. It really is a nice piece of kit though. The failsafe logging is especially brilliant and simple.
systemd covers way more ground than the traditional init system, which is why one should refer to systemd as an ecosystem of tools that come together to form a coherent experience. Many people have a problem with this, and rightfully so - I don't know what the initial plan for systemd is, but I don't doubt RedHat wanted to bundle up all the necessary tools so that system administrators didn't have to go out of their way to install each and every component.
SystemD really was developed with sysadmins in mind. Ignoring the fact that you no longer need tons of boilerplate to make service management bearable, it makes securing processes much easier. Entire classes of security issues can be wiped out or rendered harmless with just a few lines in a .service file (e.g.
PrivateTmp,NoNewPrivileges,IPAddressAllow). I'd imagine these benefits remain largely unrealized by most users since most aren't too concerned with security beyond the low-hanging fruit (especially outside the Windows world), but from what I've seen, its feature creep is entirely logical for a company that depends on licensing to large companies who care very much about security.Has its scope grown too far? I don't know. To be honest, I'm not even familiar with its entire scope, I just know that what I do use (directly) I absolutely love, and I know much of what I love is either difficult or impossible to recreate with sysvinit. The binary log files I'm ok with after watching Lennart's talk on the matter.
That depends. Some parts of systemd appear to be developed with that in mind, especially the core, and after growing accustomed to systemd units and journalctl, I became frustrated enough trying out a server running alpine (non-systemd) that I replaced it with something running systemd. Others appear focused on home systems, with odd and arbitrary design choices that make them difficult or impossible to use for sysadmins on servers. I was certainly surprised, for example, when trying out systemd-homed, I tried to add a user to a group, as root, and... was asked for the user's password, and while it is now fixed, for some time, systemd-homed apparently broke SSH. I use systemd-networkd on a number of systems, both servers and laptops, because it is in principle very nice for the occasionally complex systems I have, and declarative configuration is far better than the maze of NetworkManager or the sequence of commands of ifupdown, but for systems that need to stay up, it's odd that, for some arbitrary design reason, it's essentially impossible to have systemd-networkd reconfigure a network interface (netdev), even when they are interfaces designed to be reconfigured online (eg, wireguard): the apparently official solution is to restart the daemon, which results in a service interruption.
And like most things involving a certain subset of Linux development, asking for help or clarification about any of these issues, or opening bugs, or making suggestions, can feel like asking for a barrage of toxicity.
My problem with it is simple:
Whenever i try other distros (last time was OpenSuse TW), i need to fight with systemd at least twice every week. It always has to wait for some job to finish before shutting down my laptop. Always. I'm well versed in Linux, being an user for 18+ years, so it is not a problem on my part. I keep jobs at a mininum.
So, because it almost always holds my shutdown process, having a lot of services and the fact that i find it rather cryptic/obfuscated sometimes, i always come back to Void Linux and runit.
I feel a sense of relief whenever i boot Void. The boot process doesn't even scroll the tty. It fits right there in the screen. It is brutally fast and transparent.
Systemd is faster than what we had in the past, but it is not fast anymore.
PS: i'm not against systemd by any means, nor do i hate it. This is just my experience. I just prefer runit. I'm eager to try s6 some day.
You and I have both been Linux users for roughly the same amount of time. How come you have to fight systemd so often and I don’t?
I don't know if fight is the right word, but at least one or two times i have to wait for a job to stop running before it shuts down.
It was not a process that i could just deactivate so i needed to wait 30s, sometimes more.
These things never happens with runit, the shutdown and boot are instant and like i said, runit is way more transparent to me about what it is doing and i have complete control.
I probably could figure out why it hangs when it hangs, but i would have to dig.
I just prefer the simplicity of Void these days so i stopped trying other distros.
I’ve had what you describe as well. But in my experience this is caused by deadlocks in the filesystem.
Did you run some drive tests? Because in my case it was a symptom of a dying ssd.
I didn't at the time. I'm using the same SSD with Void now for a year with no problems.
I am currently experiencing this on my Arch+KDE+Wayland setup.
If I just click shut down, it will take (originally) 1min 30sec to stop jobs (configurable via
/etc/systemd/system.conf,DefaultTimeoutStopSec).In my experience, closing Firefox and Steam before shutting down seem to sometimes help, but it's not consistent.
I'm not sure if this is the fault of systemd or something else (Firefox? Steam? idk), but that timeout is way too long.
I never have apps open when shutting down, i always close manually, so the wait time happened even with nothing open.
It was not every time, but at least once or twice every week.
I don't mean to doubt you, but this isn't an experience I've heard people report. It's not been my experience in many years of running Ubuntu and Arch, nor is it my experience now with NixOS. Maybe you're just unlucky?
Fwiw, I do experience long shutdowns using systemd and have for years across many distros.
Interesting! I wonder if it's a hardware difference?
Might be a config thing. For example, waiting for a network mount.
It could be, but this happens to a lot of people. When i complained about it on reddit there was a lot of upvotes and people relating. I guess systemd devs don't care much about shutdown time and that's ok because it is not something you do often.
This happens with me on fresh installs with nothing out of the ordinary. No network mount.
I've had it on lots of different configs, but that's definitely possible!
Probably. Or it could be something simple, but i find systemd too cryptic and when i start digging i end up losing interest.
Runit is just so simple and it works so i'm happy with it.
As others have explained, many people had issues with the technical side of systemd. But on top of this: at the time when there was a lot of community push-back, systemd was seen to be somewhat "forced" into being the de-facto standard (against what "the community" believed to be the majority opinion) by some heavy-handed political manoeuvring. This, of course, made the controversy much worse.
Edit: add "seen to be"