Embed Notice

@patrickcmiller That's the key point I'm trying to make. Sometimes people bolt on a proxy in front of a vulnerable web site to mitigate SQL injection risk. We often criticize that as insufficient on its own. And that is exactly what they're doing to prevent information leakage out of LLMs: stick some stuff in the prompt to make undesirable data less likely in the output. And while bolt-on security can be PART of a multi-prong security approach, in the case of LLMs it's the only prong.

This picture from this blog is really useful at showing how our mental models about LLMs can be wrong. On the left is what the user thinks is happening. On the right is what is happening in the LLM. The effectiveness of the bolt-on security is limited by the fundamental workings of the technology.