Notices by jonny (nonvenomous) (jonny@neuromatch.social)

So rsync rewriting all the tests puts the entire project in play. Now the entire protective surface has been sloshed through a layer of probability, so the loop must accelerate. Followup PRs add more carveouts with lengthy LLM justifications that sound perfectly plausible but amount to an erosion of the protective surface. We go from cumulative improvement to a random walk.

i love gambling. i have used "AI" extensively. it feels the same.

So, look. One shot rewriting the whole test suite in another language is probably not great to do, but what happened here is so much worse than you are expecting.

https://github.com/RsyncProject/rsync/pull/903/

This does not "translate tests into pytest" or a unit testing framework, it writes its own testing framework where tests are whole python scripts that redefine basic test functions in every script. Surely there would be a single way to "run rsync and get the results" - nope, well, there is, but then every test file will randomly redefine its own _run_and_capture function. So like now rsync needs a test suite for its test suite.

If instead of telling an LLM to "rewrite the tests in python" you just searched "python testing" you would find the pytest docs. And then you would find examples. And then you could write fixtures to deduplicate all the prior shell script setup and teardown stuff, and so on. But since it was just "rewrite the tests in python" its now worse than before, and the odds of the rewrite actually being a 100% faithful translation are close to 0.

I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding.

Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.

LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides.

The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.

To a person, the whole purpose of the test is for it to fail when it should. That's an elemental part of writing good tests: they must fail before the patch, or else they provide no protection. We want protection from failure, that is good for us. We need tests to protect us because we can't possibly evaluate all the other parts of a complex system when we try to fix one part of it.

LLM slot machines change what tests mean - of course we still want the code to work good, but if we're not evaluating the code or the tests, then what the slot machine turns them into is just a high score and the jackpot condition. 130 new tests added, that means its good. They pass, that means I win.

The bugfix loop with LLMs defeats the purpose of automated tests and renders it no better than manual testing: you notice a bug, you yell at the LLM to fix it, you keep looking at the specific thing that's broken until its fixed, good robot, ship it. The changes don't have meaningful tests, and nothing else does either, so the slot machine loop repeats, bug->fix->win. Very velocity. Rocket fuel even.

Here's an example from some code that was thrust at me this week. The rest of the tests try a bit harder to look like tests, but this one is perplexing.

What does it test? The function name suggests its a smoke test. LLMs love to call things smoke tests. That would suggest this would be an early-run test that fails loudly if some basic precondition - like having ffmpeg - fails. Or, I guess we are smoke testing the ensure_ffmpeg function? Anyway who knows. However we first check if ffmpeg or ffprobe are present, which is exactly what ensure_ffmpeg does. If they aren't present, a warning tells us that ffmpeg/ffprobe are required for the video tests, which makes it seem like this should be a parameterizing test that controls which tests are run, which of course it does not do.

So the test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.

But its not just as simple as "OK if I read the tests I should be fine" because LLM code is often untestable. It writes code with function and class names that make it seem like a something does something, but they might just be flat wrong. Or there is some invisible fallback condition the LLM encountered while generating code and added to just make tests pass, but has entirely different behavior.

If you've watched an LLM generate a project over time, you see it generating its own private language, and ive even seen it reinvent language features like function definitions themselves. Its names form part of an increasingly inaccessible web of meaning that no human can penetrate.

Writing tests requires a kind of "information gap" where you can have enough intuition about what something does, but not how it does it, so you can a) know what it should do, b) make a strong assertion about that expectation, c) without mirroring the internal implementation's limits. That's hard! And really only possible when the foundation, (a) is true. Code must have an articulable purpose in order to be testable, that's tautological, that defines what failure is. But since LLM code increasingly detaches from any kind of stable description or expectation, even if the tests look very rigorous, you can't know if they are just tailored to the specific internal details of its function to eke out a pass, because it's hard to know what it should do anyway.

So really you have to read the test code, the code under test, and also all the other code that might call the code under test. Aka you have to read everything. And rather than reading something that was written to be read, you're wading through a slop swamp. So you can't. It takes more time than just writing it. The erosion of testing is just an intrinsic part of the loop that you can't escape without breaking the spell of the slot machine, and it is what drives the loop.

RE: https://hails.org/@hailey/116657391001259044

all the criticism has been said, all the takes been had. the only metaphor i have been finding consistently useful for understanding what is happening with people and "AI" is addiction, and specifically gambling addiction.

@tchambers @laurenshof like having a fancy diplomacy dashboard or whatever is cool, but we already do all of that. we just talk to each other. i know the people on different instances and talk to them when we have a problem. it isn't visible but that's just because it's possible to talk about things in private without it being funneled through a relay. we talk with all our members about their reports, facilitate communication between instances when needed, etc. i can't imagine a more fragmented and libertarian system of federation than one where everyone is their own data island, to call that solidarity is a gross distortion of what that term means.

@tchambers @laurenshof I hope it's clear that i am not an atproto hater at this point, and am instead a big believer of Rudy's 'crabs in a barrel' characterization - our enemies are the info giants, not each other. but i don't think that this characterization of weaknesses of the AP fediverse (of which there are many) is that useful.

@tchambers @laurenshof have you administered a fedi server? we are talking about moderation all the time with different instances, it's the first thing we do when there is a moderation need. the caricature of authoritarian big servers getting in tiffs and defederating is a really poor representation of how small servers actually interact. suggesting that all fediverse instances are big brother servers that provide infrastructure without providing agency belies the existence of many volunteer-run and cooperatively managed instances.

@jackperkins
"There's openAI and anthropic, google is still in the race, and good lord what in the fuck is going on over there at meta"

@jackperkins

RE: https://mastodon.social/@MikeElgan/116628156172886406

Canon behavior from meta, tbh

I don't care of both of the numbers might be undercounts and all the double spending isn't unfurled on that site. Meta would always have shown up as "boondoggle" on the graph. The only thing it knows how to do is boondoggles.

Context there was contrasting meta's plans which it describes as making consumer AI that helps people with daily things, and for some people those daily things are curing disease, and for others its shopping. But the answer was in response to an investor asking about shopping because uh that's the thing that makes money. This strategy sorta makes sense given their consumer surfaces, but at the end of the line of reasoning where you plug the surge protector back into itself, you gotta wonder how many people are going to find renting a shopping bot from the ad company to shop in the ad company's store based on the ad companies ads a little cyclical.

Zuck, 2026 Q1 earnings call: "our plan is to make an AI that's really good at shopping"

"The last digital capital revolution was targeted ads. The next digital capital revolution is to rent a robot to look at the ads"

Love how all this is being subsidized by the federal government resulting in a negative effective tax rate for some of the largest companies on earth.

So this was clear from the very start, that structurally the business model relies on making the web an incomprehensible mess to force chatbot usage as the only remaining viable source of information. But now that we're in the "agentic" swing of things, that language is just becoming more or less text in the ad copy: "get your agent to go out there in the wasteland of the former web and get you answers" (not mentioning who made the web into a wasteland)