Super scary stuff. Then you have to wonder what's in the proprietary software that no one ever finds. The whole thing makes me want to quit developing and live in a cave on a mountain somewhere.
Super scary stuff. Then you have to wonder what's in the proprietary software that no one ever finds. The whole thing makes me want to quit developing and live in a cave on a mountain somewhere.
The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.
Heck, the same happens on the other side of the equation: even with Big Tech staffing and money, if you have a library that almost never needs any attention, the “ownership” of that code becomes pretty theoretical too. You probably can’t build a career on being very familiar with some boring, old dependency that’s just taken for granted by everyone else.
Late to the party, but I wanted to quote from the Cathedral and the Bazaar: That essay’s older than me! These aren’t even new ideas/findings. But it hasn’t gotten easier to apply them in practice...
After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.
We’ve learned since then that new maintainers can be trouble. I think declaring that it’s your last release and freezing it should be the default choice. Any new maintainer should create a fork,...
We’ve learned since then that new maintainers can be trouble. I think declaring that it’s your last release and freezing it should be the default choice. Any new maintainer should create a fork, and they don’t need anyone’s permission to do that.
Feelings can play a part of it, especially for projects that never left the unpaid/hobby phase (like xz itself!). You don't want to see it end, so you'd rather pass the baton to someone else. A...
Feelings can play a part of it, especially for projects that never left the unpaid/hobby phase (like xz itself!). You don't want to see it end, so you'd rather pass the baton to someone else.
A fork feels like a new project, even if it's the same codebase.
This blog post gives some larger context that I found quite interesting. https://boehs.org/node/everything-i-know-about-the-xz-backdoor GitHub locked up the repository. I was enjoying the comments...
This blog post gives some larger context that I found quite interesting.
Holy shit! This is crazy, the coincidence of finding it now is just crazy. Hopefully Darknet Diaries will do a show on it. The time and effort that has gone into getting this exploit into the...
Holy shit! This is crazy, the coincidence of finding it now is just crazy. Hopefully Darknet Diaries will do a show on it. The time and effort that has gone into getting this exploit into the stack, is insane.
The analysis is also so very impressive. This is so in depth, already uncovering the intricate setup to be finally able to ship the backdoor. It’s exciting and horrifying at the same time.
That detail actually reminds me of the book The Cuckoo’s Egg by Clifford Stoll. The book is non-fiction, and it recounts Clifford’s accidental discovery of early computer espionage because a lab...
That detail actually reminds me of the book The Cuckoo’s Egg by Clifford Stoll.
The book is non-fiction, and it recounts Clifford’s accidental discovery of early computer espionage because a lab manager wanted him to work out a 75 cent error in the computer time budget at Berkeley’s computer lab. It’s a short and fun read!
Cliff Stoll's other claim to fame is selling glass Klein bottles and Mobius strip scarves. They are all guaranteed to have zero volume and will not be orientable. The packing slips have excellent...
Cliff Stoll's other claim to fame is selling glass Klein bottles and Mobius strip scarves. They are all guaranteed to have zero volume and will not be orientable. The packing slips have excellent mathemagical humour.
Nobody else had raised concerns, and I don’t believe any existing security tooling or processes would have caught this (I realise there will be a torrent of vendors claiming they detect this… but they will detect this now that somebody told them).
How advanced was the threat actor? The backdoor attempt was a very serious one, with a very high bar of knowledge, research, development and tradecraft to reach this far into the Linux ecosystem. Additionally, changes made by the threat actor on Github span multiple years, and include things like introducing functions incompatible with OSS Fuzzer due to outstanding small issues since 2015, then getting OSS Fuzzer to exclude XZ Utils from scanning last year. The backdoor itself is super well put together, and even includes the ability to remotely deactivate and remove the backdoor via a kill command. Several days in, despite global focus, I haven’t seen anybody who has finished reverse engineering it.
Also, Andres had a unique testing environment and a set of coincidental setup issues which allowed him to discover the issue. I don’t know of anybody else has this setup.
When I installed a vulnerable Linux box, I had to double check it was actually vulnerable as I wouldn’t even see a speed issue. For me, it was a completely transparent backdoor — where sshd was running from disk as usual, with the usual file hash and no extra network activity.
And a detailed an well explained analysis of the shell scripts/... injecting the malware into the build process from the same blog: https://research.swtch.com/xz-script
And a detailed an well explained analysis of the shell scripts/... injecting the malware into the build process from the same blog: https://research.swtch.com/xz-script
With the current mistrust against liblzma and its complex code, I wonder if one could write a simple drop-in replacement for it. Perhaps a focus on simple code over performance might also result...
With the current mistrust against liblzma and its complex code, I wonder if one could write a simple drop-in replacement for it. Perhaps a focus on simple code over performance might also result in somewhat usable performance too, like Serenity's Ladybird Browser.
There's the lzip format, which is is supposed to be that. Focused on simplicity and security, lzip is supposed to be a drop-in replacement for gzip with better compression ratio (via the LZMA...
There's the lzip format, which is is supposed to be that. Focused on simplicity and security, lzip is supposed to be a drop-in replacement for gzip with better compression ratio (via the LZMA algorithm) and a more modern file format. It was introduced because of weaknesses in the .xz-format, but from what I can tell, it is not very popular; most users either seem to continue to use .xz or have switched to zstd.
Good technical detail in a followup Ars Technica article: https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/ . Very grokkable by...
Another post with more findings and a list of affected distros and versions: https://jfrog.com/blog/xz-backdoor-attack-cve-2024-3094-all-you-need-to-know/
Reading about this since the start, the main thing in my mind is: Imagine if the work by JT75 was spread among several users: It would be impossible to undo all of it. Of course, it is most likely...
Reading about this since the start, the main thing in my mind is: Imagine if the work by JT75 was spread among several users: It would be impossible to undo all of it.
Of course, it is most likely that the above is true: any organized bad actor would do so. Which would imply that mostly everything is compromised (even double compromised).
Super scary stuff. Then you have to wonder what's in the proprietary software that no one ever finds. The whole thing makes me want to quit developing and live in a cave on a mountain somewhere.
Just keep your backups there and visit a couple times a year! :)
Some commentary:
Technologist vs spy: the xz backdoor debate
Late to the party, but I wanted to quote from the Cathedral and the Bazaar:
That essay’s older than me!
These aren’t even new ideas/findings. But it hasn’t gotten easier to apply them in practice anyhow…
We’ve learned since then that new maintainers can be trouble. I think declaring that it’s your last release and freezing it should be the default choice. Any new maintainer should create a fork, and they don’t need anyone’s permission to do that.
That makes total sense. At the risk of opening a can of worms, why isn’t this the norm?
Feelings can play a part of it, especially for projects that never left the unpaid/hobby phase (like xz itself!). You don't want to see it end, so you'd rather pass the baton to someone else.
A fork feels like a new project, even if it's the same codebase.
This blog post gives some larger context that I found quite interesting.
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
GitHub locked up the repository. I was enjoying the comments on that binary commit too.
https://github.com/tukaani-project/xz/
Holy shit! This is crazy, the coincidence of finding it now is just crazy. Hopefully Darknet Diaries will do a show on it. The time and effort that has gone into getting this exploit into the stack, is insane.
The analysis is also so very impressive. This is so in depth, already uncovering the intricate setup to be finally able to ship the backdoor. It’s exciting and horrifying at the same time.
XKDC - 2347 exploit
Lol love that xkdc. Also funny how this guy found it. His ssh logins taking 500ms longer
That detail actually reminds me of the book The Cuckoo’s Egg by Clifford Stoll.
The book is non-fiction, and it recounts Clifford’s accidental discovery of early computer espionage because a lab manager wanted him to work out a 75 cent error in the computer time budget at Berkeley’s computer lab. It’s a short and fun read!
Cliff Stoll's other claim to fame is selling glass Klein bottles and Mobius strip scarves. They are all guaranteed to have zero volume and will not be orientable. The packing slips have excellent mathemagical humour.
Another write-up by Kevin Beaumont:
https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd
Most detailed timeline I have found so far:
https://research.swtch.com/xz-timeline
And a detailed an well explained analysis of the shell scripts/... injecting the malware into the build process from the same blog: https://research.swtch.com/xz-script
With the current mistrust against liblzma and its complex code, I wonder if one could write a simple drop-in replacement for it. Perhaps a focus on simple code over performance might also result in somewhat usable performance too, like Serenity's Ladybird Browser.
There's the lzip format, which is is supposed to be that. Focused on simplicity and security,
lzipis supposed to be a drop-in replacement forgzipwith better compression ratio (via the LZMA algorithm) and a more modern file format. It was introduced because of weaknesses in the .xz-format, but from what I can tell, it is not very popular; most users either seem to continue to use.xzor have switched tozstd.Good technical detail in a followup Ars Technica article: https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/ . Very grokkable by average devs.
Another post with more findings and a list of affected distros and versions:
https://jfrog.com/blog/xz-backdoor-attack-cve-2024-3094-all-you-need-to-know/
Reading about this since the start, the main thing in my mind is: Imagine if the work by JT75 was spread among several users: It would be impossible to undo all of it.
Of course, it is most likely that the above is true: any organized bad actor would do so. Which would imply that mostly everything is compromised (even double compromised).