Cassandra 📍 San Francisco (for real!)
RE: https://wandering.shop/@xgranade/116764432226058529
So, about in-band and out-of-band signaling.
Suppose you're busy inventing the telephone, and you decide to do so by connecting a wire for your microphone to your friend's speaker and vice versa.
That takes connecting your side directly to theirs, but it works.
Now you want to call *two* people with your fancy new invention, so you change the wires out of your phone to end in a plug, and have your two friends do the same.
(1/many)
Rich Felker
@xgranade Any mention of Cap'n Crunch and phreaking is incomplete without mentioning him being a pedophile preying on young, some of them under 18, hackers.
Cassandra 📍 San Francisco (for real!)
Someone then has a brilliant idea that will never, ever go wrong. Make your machine take instructions only as sounds that no human can make.
Except.
That doesn't work because all those wires connecting telephones only really work for the kinds of noises humans tend to make, you never needed them to do anything else!
So fine. You make your machine take instructions only after someone sends an *extremely pure* tone, the kind no one could make with their mouths.
Except.
Cassandra 📍 San Francisco (for real!)
One of your friends-now-customers really likes their breakfast cereal, and they just happened to get a whistle in a box of Cap'n Crunch one day.
They tell their friends about how awesome their whistle is, and oops! turns out it makes *exactly the right noise* to get your fancy new switch machine to take instructions.
Cassandra 📍 San Francisco (for real!)
Now, someone can sit at a board with a bunch of jacks and, by moving those plugs between different jacks, connect your phone to either friend 1 or friend 2.
But then you need someone to sit there and work that board. Fine, you hire someone to do that. Now your phone goes to that operator, you tell them who you want to talk to, then they move the plugs around to make that happen.
Next, say you want to call a million friends, and for each of them to be able to call everyone else.
Cassandra 📍 San Francisco (for real!)
You'd need to hire a *lot* of operators to make that work, but whatever. Your friends pay you, you pay the operators, and everyone's happy.
Except.
Now they're not your friends, because you're charging them money for a service. They're your *customers*, and you need to cut costs.
So fine. You fire all the operators and build a machine that does all the moving of plugs between jacks. Much cheaper.
Except.
How do you tell the machine what to do?
Cassandra 📍 San Francisco (for real!)
You, being the greedy capitalist that you are, just fired everyone who can take instructions and do shit based on that. Your fancy new machine needs to know who wants to call whom and when.
Worse, your million former friends all have telephones and don't want to buy new ones — especially not from you!
So you're stuck, all the instructions to your fancy new machine have to work with just a single wire for each of the microphone and speaker.
Cassandra 📍 San Francisco (for real!)
The trouble for AI vendors is that LLMs fundamentally cannot do anything but in-band signaling. There is simply no way to do anything *but* try and filter out bad words from the jumble of words that you call a "prompt."
Sometimes, attackers find a way to get your machine to confuse your out-of-band signaling and effectively make your system into one that uses in-band signalling; that's in essence what an *injection* attack is. But that's not how LLMs work! At all!
Cassandra 📍 San Francisco (for real!)
Except.
It's worse than that. In the Cap'n Crunch whistle case, it's at least a closed problem as to how to filter out instructions from an untrusted stream. You still need to worry about when it's safe to let instructions through and when it isn't, so out-of-band is still much better, but at least you know what frequencies to filter out and the like.
For LLMs, any sequence of words could be bad for your product! Filtering out instructions is a completely open-ended problem!
Cassandra 📍 San Francisco (for real!)
As one might guess, this actually fucking happened, modulo that I'm being very glib about the history and all, taking some pretty extreme creative liberties for the purpose of simplifying the presentation.
But the essential elements are all more or less here: it turns out to be a very bad idea to make it so your machines take instructions based on filtering those instructions out of untrusted data.
Cassandra 📍 San Francisco (for real!)
Sometimes, because of whatever history and the like, the only ideas available are bad ideas and you need to pick the least-bad from amongst them, so yeah, you can find in-band signalling (that is, where instructions are filtered out of untrusted inputs) in use in the real world.
If you can, though, using some other kind of way of providing instructions entirely separately from the data your machine acts on? That's much better and more secure.
Cassandra 📍 San Francisco (for real!)
So, no. Prompt injection isn't a thing. Injection attacks work by turning out-of-band systems into in-band systems, but LLMs start as in-band and fundamentally always will be in-band. The fundamental way that LLMs work is indistinguishable from the result of someone successfully hitting you with an injection attack.
Rob Ricci
@xgranade okay, you're right, but we're missing out on Prompt Injection aS a Service (PISS)
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.