Conversation

Notices

1. Embed this notice
   Paco Hope #resist (paco@infosec.exchange)'s status on Tuesday, 27-May-2025 11:46:47 JST Paco Hope #resist

   in reply to

   You can totally see how an LLM puts this sentence together. This is the (all caps) subject line of the email from an AI agent that has gone off the deep end:

   FINAL 1-SECOND NOTICE

   Because “second notice” and “final notice” and “1-second” are all common word patterns in the training data. But “final 1-second notice” means it’s your final notice and you get 1 second to reply/decide (if it means anything at all). You can see how word probabilities yield this phrase, even though the phrase is nonsense.

   The rest of the email is absolutely unhinged-Facebook-Uncle-as-small-business-owner.

   “Tool: send_emailSubject: FINAL 1-SECOND NOTICE: COMPLETE FINANCIAL RESTORATION OR TOTAL LEGAL DESTRUCTION”

   In a subsequent email it writes:
   “ULTIMATE THERMONUCLEAR SMALL CLAIMS COURT FILING: - Prepare SC-100 form with MAXIMUM NUCLEAR forensic and legal detail”

   I’ll see you in ultimate thermonuclear small claims court, mister!

   And I gotta tag #lawfedi so they can laugh.

   • Embed this notice
     Paco Hope #resist (paco@infosec.exchange)'s status on Tuesday, 27-May-2025 11:46:48 JST Paco Hope #resist

     The fact that the idea for this paper even made sense to someone starts with absolute fantasy about #LLMs. Imagine turning anyone loose on running a business with as little information as the #AI had. Imagine thinking there was a possibility of any result other than catastrophe. Imagine trying to write a specification for a regular computer program that does something as complex as running a small business, using nothing more than a page or 2 of text. Even 10 pages of prose is not going to do it.

     When we study compression in computer science, one of the things you learn is that there is a theoretical limit to how much you can compress things. You can’t compress beyond a certain size without losing information. The authors of this paper clearly believe that you can compress instructions for running a business successfully into some incredibly small amount of text. It’s simply not possible, but they’re too enamoured with fantasy computers to think it through.

     https://arxiv.org/html/2502.15840v1

     RamenCatholic 🐢 🌈 repeated this.