Conversation

Notices

1. Embed this notice
   Patrick C Miller :donor: (patrickcmiller@infosec.exchange)'s status on Friday, 18-Oct-2024 20:12:02 JST Patrick C Miller :donor:

   10 most critical LLM vulnerabilities https://www.csoonline.com/article/575497/owasp-lists-10-most-critical-large-language-model-vulnerabilities.html

   • Embed this notice
     noplasticshower (noplasticshower@infosec.exchange)'s status on Friday, 18-Oct-2024 20:33:02 JST noplasticshower

     in reply to

     @patrickcmiller not good work

   • Embed this notice
     Paco Hope #resist (paco@infosec.exchange)'s status on Friday, 18-Oct-2024 21:14:03 JST Paco Hope #resist

     in reply to

     @patrickcmiller They left out the authorisation model. All #AI systems I have seen have a binary authorisation model: an entity is allowed to inference against the model, or not. Contrast with relational databases where you can have access to some tables and not others. We can even get to row-level and column-level access controls. Just because you can query the database doesn’t mean the whole of the dataset is available to you. Data in the database that matches your query might be missing from your response because you don’t have access to those items.

     With an #LLM the entire trained model is available for inference. To put it in #RBAC terms, every distinct role with distinct access to subsets of data would need its own model, trained only on the data they’re allowed to access.

     In practice no one does that. So models either include too much data, risking exposure to unauthorised users, or they omit useful data in training because they don’t want the risk. Middle ground solutions are rare and difficult.

   • Embed this notice
     Patrick C Miller :donor: (patrickcmiller@infosec.exchange)'s status on Saturday, 19-Oct-2024 09:15:00 JST Patrick C Miller :donor:

     in reply to

     • Paco Hope #resist

     @paco some data in the model isn’t available though. For example, it won’t provide (or they make an attempt at restricting) dangerous or harmful information.

   • Embed this notice
     Patrick C Miller :donor: (patrickcmiller@infosec.exchange)'s status on Saturday, 19-Oct-2024 09:15:27 JST Patrick C Miller :donor:

     in reply to

     • noplasticshower

     @noplasticshower is there a better list somewhere?

   • Embed this notice
     Paco Hope #resist (paco@infosec.exchange)'s status on Sunday, 20-Oct-2024 00:36:11 JST Paco Hope #resist

     in reply to

     @patrickcmiller That's the key point I'm trying to make. Sometimes people bolt on a proxy in front of a vulnerable web site to mitigate SQL injection risk. We often criticize that as insufficient on its own. And that is exactly what they're doing to prevent information leakage out of LLMs: stick some stuff in the prompt to make undesirable data less likely in the output. And while bolt-on security can be PART of a multi-prong security approach, in the case of LLMs it's the only prong.

     This picture from this blog is really useful at showing how our mental models about LLMs can be wrong. On the left is what the user thinks is happening. On the right is what is happening in the LLM. The effectiveness of the bolt-on security is limited by the fundamental workings of the technology.

   • Embed this notice
     Patrick C Miller :donor: (patrickcmiller@infosec.exchange)'s status on Sunday, 20-Oct-2024 10:08:22 JST Patrick C Miller :donor:

     in reply to

     • Paco Hope #resist

     @paco great explanation!