Conversation

Notices

1. Embed this notice
   AndresFreundTec (andresfreundtec@mastodon.social)'s status on Saturday, 30-Mar-2024 04:44:21 JST AndresFreundTec

   I accidentally found a security issue while benchmarking postgres changes.

   If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.

   https://www.openwall.com/lists/oss-security/2024/03/29/4

   • Haelwenn /элвэн/ :triskell:, clacke and gidi like this.
   • Haelwenn /элвэн/ :triskell:, gidi and zunda and 3 others repeated this.
   • Embed this notice
     AndresFreundTec (andresfreundtec@mastodon.social)'s status on Saturday, 30-Mar-2024 04:44:20 JST AndresFreundTec

     in reply to

     I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.

     Really required a lot of coincidences.

     Haelwenn /элвэн/ :triskell:, clacke and gidi like this.
     Haelwenn /элвэн/ :triskell: and AnthonyJK-Admin repeated this.
   • Embed this notice
     TTimo (ttimo@mastodon.social)'s status on Saturday, 30-Mar-2024 05:57:21 JST TTimo

     in reply to

     @AndresFreundTec jesus, I hope you like beer cuz we owe you a free lifetime supply.

     Haelwenn /элвэн/ :triskell: likes this.
     AnthonyJK-Admin repeated this.
   • Embed this notice
     John Regehr (regehr@mastodon.social)'s status on Saturday, 30-Mar-2024 06:55:18 JST John Regehr

     in reply to

     @AndresFreundTec every internet user owes you a beer

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     silverwizard (silverwizard@convenient.email)'s status on Saturday, 30-Mar-2024 08:14:06 JST silverwizard

     in reply to
     @AndresFreundTec Good job! You deserve a lot of kudos!
   • Embed this notice
     Mark Eichin (eichin@mastodon.mit.edu)'s status on Saturday, 30-Mar-2024 11:43:55 JST Mark Eichin

     in reply to

     @AndresFreundTec And a lot of persistence! Reminds me of one of the classics of the industry, Cliff Stoll's Cuckoo's Egg - "Stoll traced the error to an unauthorized user who had apparently used nine seconds of computer time and not paid for it" leading to a german hacker selling content to the KGB - 38 years ago. It is impressive (but uncommon) to see someone paying that level of attention to anomalies these days, with how thick tech stacks have gotten...

     Haelwenn /элвэн/ :triskell: and clacke like this.
   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Saturday, 30-Mar-2024 16:27:16 JST phryk 🏴

     in reply to

     @AndresFreundTec Thanks a whole bunch, excellent work and just in the nick of time to avert the worst consequences. <3

     clacke likes this.
   • Embed this notice
     Arik (arikb@mastodon.sdf.org)'s status on Saturday, 30-Mar-2024 16:27:23 JST Arik

     in reply to

     @AndresFreundTec It's not coincidence - it's attention and perseverance. The Internet thanks you.

     clacke likes this.
   • Embed this notice
     Glyph (glyph@mastodon.social)'s status on Saturday, 30-Mar-2024 16:27:27 JST Glyph

     in reply to

     @AndresFreundTec I feel both confident and also kind of queasy when saying this: it seems extremely likely that this is not the first time something like this has happened, it's just the first time we have been lucky enough to notice.

     clacke likes this.
   • Embed this notice
     Ⓐ Dirk Ritter (gnumatic@deppenkessel.de)'s status on Sunday, 31-Mar-2024 05:07:43 JST Ⓐ Dirk Ritter

     in reply to

     • TTimo

     @TTimo @AndresFreundTec

     According to the test script fixed in testing as of today. Great. 👍

     (As in maybe "right now", i.e. somewhere between early morning update and now - as we're approaching midnight. Phew!)

     Beer, peanuts... 🥳

   • Embed this notice
     AndresFreundTec (andresfreundtec@mastodon.social)'s status on Monday, 01-Apr-2024 10:03:48 JST AndresFreundTec

     in reply to

     Just to be clear: I didn't mean that I didn't do good - I did. I mean that we got unreasonably lucky here, and that we can't just bank on that going forward.

     Haelwenn /элвэн/ :triskell:, clacke and Linux Walt Alt (@lnxw37a2) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} like this.
   • Embed this notice
     AndresFreundTec (andresfreundtec@mastodon.social)'s status on Monday, 01-Apr-2024 10:03:49 JST AndresFreundTec

     in reply to

     Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.

     Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.

     Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().

     clacke likes this.
   • Embed this notice
     AndresFreundTec (andresfreundtec@mastodon.social)'s status on Monday, 01-Apr-2024 10:03:49 JST AndresFreundTec

     in reply to

     There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.

     clacke repeated this.
   • Embed this notice
     AndresFreundTec (andresfreundtec@mastodon.social)'s status on Monday, 01-Apr-2024 10:03:50 JST AndresFreundTec

     in reply to

     One more aspect that I think emphasizes the number of coincidences that had to come together to find this:

     I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.

   • Embed this notice
     Christoph Petrausch (hikhvar@norden.social)'s status on Monday, 01-Apr-2024 10:11:52 JST Christoph Petrausch

     in reply to

     @AndresFreundTec https://norden.social/@hikhvar/112180483133159589

     clacke likes this.
   • Embed this notice
     LisPi (lispi314@udongein.xyz)'s status on Monday, 01-Apr-2024 10:12:02 JST LisPi

     in reply to

     • Glyph
     • 🍒🌳 Hartmut Goebel
     @kirschwipfel @glyph @AndresFreundTec > If the packager chooses to use the official tarball as "the source", validating the checksum would not have helped. :-(
     
     Unfortunately, yeah.
     
     > Also whether it's always possible to run running autoreconf depends on the content of the tarball.
     
     Of course if it isn't a C project then it probably isn't. If it is such a project, then one should have such tooling installed.
     
     > Which brings me to the (preliminary) conclusion that we'd better use repos as source of trust
     
     That is more sensible generally, as the history of an object and its belonging to a project is a reified (and verifiable) relationship under code versioning sytems, unlike arbitrary buckets of files.
     clacke likes this.
   • Embed this notice
     🍒🌳 Hartmut Goebel (kirschwipfel@nerdculture.de)'s status on Monday, 01-Apr-2024 10:12:03 JST 🍒🌳 Hartmut Goebel

     in reply to

     • Glyph
     • LisPi

     > I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.

     If the packager chooses to use the official tarball as "the source", validating the checksum would not have helped. :-( Also whether it's always possible to run running autoreconf depends on the content of the tarball.

     Which brings me to the (preliminary) conclusion that we'd better use repos as source of trust
     @lispi314 @AndresFreundTec @glyph

   • Embed this notice
     LisPi (lispi314@udongein.xyz)'s status on Monday, 01-Apr-2024 10:12:04 JST LisPi

     in reply to

     • Glyph
     @glyph @AndresFreundTec That is true.
     
     Binary artifacts have no business existing in Free Software (or near-binary considering how auditable pre-generated config scripts end-up being). The way it was compromised in this case is almost certain to have happened before and reminds me of the SourceForge malware debacle (so arguably that's another famous example of it happening before).
     
     I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
   • Embed this notice
     LisPi (lispi314@udongein.xyz)'s status on Monday, 01-Apr-2024 10:12:06 JST LisPi

     in reply to

     • Glyph
     • Martin Ueding
     • 🍒🌳 Hartmut Goebel
     @martin_ueding @glyph @AndresFreundTec @kirschwipfel > Sounds good. But some projects have a build stage which generates lots of things. Packagers for distributions need to up the needed environment and perform these steps. It seems much easier to use a provided artifact in these cases.
     
     If you don't care about reliability, build-reproducibility and trustworthiness of your distro, sure.
     
     > I think this attack is hard to defend against: An evil insider in the project with control over the code and artifacts.
     
     Compromise of artifacts is a lot easier, you can ask the Linux Mint project about that.
     
     > One could also hide malicious stuff in the code itself directly, in plain sight.
     
     That requires a project with such awful code practices (accepting random unauditable blobs) that you'd hope the package maintainers would simply refuse to add it in the first place.
     clacke likes this.
   • Embed this notice
     Martin Ueding (martin_ueding@bonn.social)'s status on Monday, 01-Apr-2024 10:12:09 JST Martin Ueding

     in reply to

     • Glyph
     • 🍒🌳 Hartmut Goebel
     • LisPi

     @kirschwipfel @lispi314 @AndresFreundTec @glyph
     Sounds good. But some projects have a build stage which generates lots of things. Packagers for distributions need to set up the needed environment and perform these steps. It seems much easier to use a provided artifact in these cases.

     I think this attack is hard to defend against: An evil insider in the project with control over the code and artifacts. One could also hide malicious stuff in the code itself directly, in plain sight.

   • Embed this notice
     gokuldas (gokuldas@mastodon.social)'s status on Tuesday, 09-Apr-2024 16:24:05 JST gokuldas

     in reply to

     @AndresFreundTec "... and kids, that's how i saved the world"

     clacke likes this.
   • Embed this notice
     Gordon Messmer (gordonmessmer@fosstodon.org)'s status on Tuesday, 09-Apr-2024 16:24:07 JST Gordon Messmer

     in reply to

     @AndresFreundTec Thank you so much for finding this!

     The questions at the top of my mind now are: who will fork and continue maintenance of xz? How will we determine that we can trust them? And how will we apply those lessons throughout the larger ecosystem?

     clacke likes this.
   • Embed this notice
     Gordon Messmer (gordonmessmer@fosstodon.org)'s status on Tuesday, 09-Apr-2024 16:24:11 JST Gordon Messmer

     in reply to

     • malte

     @malte @AndresFreundTec There's still a lot of data out there already in xz format, so merely dropping the software would mean that that data becomes unreadable. Dropping it may be an option, but I'm not sure it's the best option.

     clacke likes this.
   • Embed this notice
     malte (malte@anticapitalist.party)'s status on Tuesday, 09-Apr-2024 16:24:12 JST malte

     in reply to

     • Gordon Messmer

     @gordonmessmer @AndresFreundTec i read that many projects migrate to zstd anyway ¯\_(ツ)_/¯