Conversation

Notices

1. Embed this notice
   scriptjunkie (sj@social.scriptjunkie.us)'s status on Thursday, 06-Aug-2026 22:38:06 JST scriptjunkie

   Can we stop feigning shock that AI labs' security engineers didn't catch models hacking? The whole job description has always been to catch outsiders hacking your own stuff. 99% of security teams never even consider "catch us hacking others" which is a very different job.

   • Embed this notice
     Chris Bohn (docbohn@techhub.social)'s status on Friday, 07-Aug-2026 00:33:27 JST Chris Bohn

     in reply to

     @sj Sometimes it's both. (story from pre-LLM era)

     We had an IT support contractor who was the odds-on favorite to win the follow-on contract (as incumbents typically are).

     One day (thanks to a whistleblower) we discovered they were an insider threat: we had another contract up for bid that they had no experience in, but they wanted to compete for. So they tasked a few people with copying our copies of *that* incumbent's files, so they could copy the processes and look like they knew what they were doing.

     No one wanted this contractor to have the follow-on IT support contract, but under federal acquisition regulations we had to use the rubric, and they kept ticking box after box. Finally, I pointed out that the contract requires notifying us within 48 hours of detecting an intrusion.

     "Logically, they detected their own intrusion immediately upon performing the intrusion. They didn't 'notify' us until we called them on it a week later. Therefore, they have demonstrated that they cannot satisfy at least one part of the contract."