Notices by Cassandra 📍 San Francisco (for real!) (xgranade@wandering.shop), page 2

unvalued goose game

https://spacey.space/@pauldrye/116811109218038661

No valued goose is commemorated here.

So, no. Prompt injection isn't a thing. Injection attacks work by turning out-of-band systems into in-band systems, but LLMs start as in-band and fundamentally always will be in-band. The fundamental way that LLMs work is indistinguishable from the result of someone successfully hitting you with an injection attack.

Sometimes, because of whatever history and the like, the only ideas available are bad ideas and you need to pick the least-bad from amongst them, so yeah, you can find in-band signalling (that is, where instructions are filtered out of untrusted inputs) in use in the real world.

If you can, though, using some other kind of way of providing instructions entirely separately from the data your machine acts on? That's much better and more secure.

As one might guess, this actually fucking happened, modulo that I'm being very glib about the history and all, taking some pretty extreme creative liberties for the purpose of simplifying the presentation.

But the essential elements are all more or less here: it turns out to be a very bad idea to make it so your machines take instructions based on filtering those instructions out of untrusted data.

https://en.wikipedia.org/wiki/Phreaking

Except.

It's worse than that. In the Cap'n Crunch whistle case, it's at least a closed problem as to how to filter out instructions from an untrusted stream. You still need to worry about when it's safe to let instructions through and when it isn't, so out-of-band is still much better, but at least you know what frequencies to filter out and the like.

For LLMs, any sequence of words could be bad for your product! Filtering out instructions is a completely open-ended problem!

The trouble for AI vendors is that LLMs fundamentally cannot do anything but in-band signaling. There is simply no way to do anything *but* try and filter out bad words from the jumble of words that you call a "prompt."

Sometimes, attackers find a way to get your machine to confuse your out-of-band signaling and effectively make your system into one that uses in-band signalling; that's in essence what an *injection* attack is. But that's not how LLMs work! At all!

RE: https://wandering.shop/@xgranade/116764432226058529

So, about in-band and out-of-band signaling.

Suppose you're busy inventing the telephone, and you decide to do so by connecting a wire for your microphone to your friend's speaker and vice versa.

That takes connecting your side directly to theirs, but it works.

Now you want to call *two* people with your fancy new invention, so you change the wires out of your phone to end in a plug, and have your two friends do the same.

(1/many)

You, being the greedy capitalist that you are, just fired everyone who can take instructions and do shit based on that. Your fancy new machine needs to know who wants to call whom and when.

Worse, your million former friends all have telephones and don't want to buy new ones — especially not from you!

So you're stuck, all the instructions to your fancy new machine have to work with just a single wire for each of the microphone and speaker.

You'd need to hire a *lot* of operators to make that work, but whatever. Your friends pay you, you pay the operators, and everyone's happy.

Except.

Now they're not your friends, because you're charging them money for a service. They're your *customers*, and you need to cut costs.

So fine. You fire all the operators and build a machine that does all the moving of plugs between jacks. Much cheaper.

Except.

How do you tell the machine what to do?

Now, someone can sit at a board with a bunch of jacks and, by moving those plugs between different jacks, connect your phone to either friend 1 or friend 2.

But then you need someone to sit there and work that board. Fine, you hire someone to do that. Now your phone goes to that operator, you tell them who you want to talk to, then they move the plugs around to make that happen.

Next, say you want to call a million friends, and for each of them to be able to call everyone else.

One of your friends-now-customers really likes their breakfast cereal, and they just happened to get a whistle in a box of Cap'n Crunch one day.

They tell their friends about how awesome their whistle is, and oops! turns out it makes *exactly the right noise* to get your fancy new switch machine to take instructions.

Someone then has a brilliant idea that will never, ever go wrong. Make your machine take instructions only as sounds that no human can make.

Except.

That doesn't work because all those wires connecting telephones only really work for the kinds of noises humans tend to make, you never needed them to do anything else!

So fine. You make your machine take instructions only after someone sends an *extremely pure* tone, the kind no one could make with their mouths.

Except.

Yes, but only one of these is an "injection." SQL injection attacks work by turning what's normally out-of-band signaling into in-band... but LLMs *only* have in-band signaling. There's no concept of having separate control and data streams in the first place.

https://techhub.social/@rayckeith/116762201282703808

It's why "prompt injection" is a bad term. It implies that an attack causes control and data to become mixed, but there's no difference where LLMs are concerned. Fuck, LLMs aren't even controllable in any concrete sense, you just kinda throw shit at them and hope they do what you wanted.

Man, I am not looking forward to how many people are going to lose their retirements to the Anthropic, SpaceX, and OpenAI financial scams.

@SnoopJ https://xkcd.com/774/

Just ran across "annoying Bluesky liberal" as a phrase, and wow. Like, my dude, you have no fucking idea how annoying or how liberal it gets if you think Bluesky is too far.

Hell, I'm so annoying, I treat "liberal" as an insult that punches from the left!

RE: https://fosstodon.org/@ThePSF/116722808810621981

I am the particular mix of conflict-avoidant and high-conflict that I'll say this here rather than joining a call, but my main question at this point is how to reconcile the PSF's rejection of federal grants tied to disavowing DEI priorities with the PSF's acceptance and adoption of AI tooling that similarly comprises a rejection of DEI priorities.

On the whole, I still think Python as a community is doing better than many language communities at navigating that tension, but that doesn't mean that Python *as an institution* is living up to its own stated values.

Failing to do so is understandable, I absolutely would not do better were I in power there, so it may even be hypocritical for me to raise this at all — that said, all I'm hoping for is some external communication that acknowledges the problem.