Notices by AndresFreundTec (andresfreundtec@mastodon.social)

Hoped for effect of AI use of new postgres contributors: Faster rampup, higher quality patches.

Real effect: A torrent of pointless patches, which often don't even pass tests. A torrent of AI generated pointless review comments.

Debugging a bug where the repro takes anywhere from 8 to 40h (when run concurrently), which can only be reproduced on one platform (macos) and where adding additional debug information drives the time to reproduce up further is ... not fun.

A few more days of torturing my mac mini later:

Found the problem, a missing memory barrier in a corner case.

With the knowledge of the issue the reduced repro takes < 3s to crash.

@corbet @LWN Do you see a lot of pointlessly redundant requests? I see a lot of related-seeming IPs request the same pages over and over.

@rsc I've not seen a whole lot of progress. A few focused improvements in systemd's dependencies and openssh merged a few improvements. While good, they aren't really anything systemic.

But perhaps I'm just a pessimistic^Wrealistic grump and not seeing the full picture.

Does anybody have a recommendation for a decent read-heavy/only "OLTP-ish" database benchmark where the queries aren't just single-table point or range selects?

Would like to showcase some scalability problems + solution that I see with such queries, but right now it's mostly visible with queries I made up on my own. It's easy enough to make up workloads that show almost anything, so I'd rather use something designed by someone else.

@GossiTheDog Hah. This is weird, but not that kind of weird, I strongly suspect.

Although I wouldn't mind getting that farm.

https://gist.github.com/anarazel/ca7d1db68fb7380d21f6fd819a147df1

How can two cores on the same CPU such crazily different icache behaviour? For the same process!

Well, color me very confused. In a CPU bound workload two cores on the same socket have substantially different performance (32% slowdown). If I just migrate the running process between the cores, performance changes immediately.

This is on 2x Xeon 5215 system.

I checked that it's not thermals, cpu frequency/boost and the system is idle.

Here's the odd part: The biggest difference evident in perf counters is
a 2.5x difference in icache_64b.iftag_stall, with ~same icache_64b.iftag_miss.

The "new" Linux CVE approach seems ... unhelpful at best. I know from experience that dealing with security reports is a pain and I assume that's way worse for Linux than for Postgres. But this just seems to purely be aimed at annoying everyone.

@hyc @carnildo From what I can tell, the check for permitted algorithms is after RSA_public_decrypt() has already been called, at least on some relevant paths.

@praseodym Hah. I've apparently been doing this stuff for a while.

@bcantrill @ahl Now we just need to avoid derailing into a heated debate about solaris/illumos being good/bad...

@bcantrill @ahl There might be more agreement on that :)

I did not think that finding a security vulnerability would lead to tabloids digging around my life. Including "analyzing" (aka making things up) the one personal-ish picture I've ever shared on social media. FFS.

Oddly enough, they weren't interested in lots of kinda pretty graphs.

@GossiTheDog I think there are a few people looking into that.

If I were the team behind "jia", I'd have looked at getting into dissimilar projects, not the same project multiple times, not multiple compression libs. But of course there are other actors...

The scariest areas I can think of are, in that order, compilers / binutils, buildsystems, "build executors" like make/ninja.

I am a bit concerned by all the focus on small-ish projects with overwhelmed maintainers. There indeed are a lot of problems in that area.

But I am certain that lots of experienced OSS devs can think of a few large and crucial projects where they fairly easily could have hidden something small in a larger change. Without a lot of prior contributions to the project.

One more aspect that I think emphasizes the number of coincidences that had to come together to find this:

I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.

There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.

Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.

Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.

Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().