Conversation

Notices

1. Embed this notice
   John Pettigrew (johnpettigrew@wandering.shop)'s status on Thursday, 28-Mar-2024 20:41:41 JST John Pettigrew

   For those of you who use LLMs to help you code, here's a warning: these tools have been shown to hallucinate packages in a way that allows an attacker to poison your application. https://www.theregister.com/2024/03/28/ai_bots_hallucinate_software_packages/ #ai #gpt #chatgpt #security

   • Tim Chambers repeated this.
   • Embed this notice
     Erik Jonker (erikjonker@mastodon.social)'s status on Thursday, 28-Mar-2024 23:24:39 JST Erik Jonker

     in reply to

     @johnpettigrew ...true, but there also some tasks, like analysing code (on various aspects), which have quite a lowrisk, although you can still not rely on it ofcourse

     GreenSkyOverMe (Monika) repeated this.
   • Embed this notice
     splatt9990 (splatt9990@fosstodon.org)'s status on Thursday, 28-Mar-2024 23:24:39 JST splatt9990

     in reply to

     • Erik Jonker

     @ErikJonker @johnpettigrew
     Actually, encouraging that use case is kind of dangerous due to the automation bias

     https://en.wikipedia.org/wiki/Automation_bias

     Essentially, after a while, even a good programmer will start to become less and less critical of the LLM's output, even when they shouldn't be. This has the potential to lead to very serious bugs. You could probably catch this with a very thorough code review step but good code review is hard to do.

   • Embed this notice
     John Pettigrew (johnpettigrew@wandering.shop)'s status on Thursday, 28-Mar-2024 23:24:40 JST John Pettigrew

     in reply to

     • Erik Jonker

     @ErikJonker This seems to be the live question: which subset of tasks that coders do are actually made quicker or better by using LLMs, given that you have to thoroughly check every character and every step of logic in what they suggest. I know several folk who swear by it but, personally, I'm too much of a control freak to take the chance.

   • Embed this notice
     Erik Jonker (erikjonker@mastodon.social)'s status on Thursday, 28-Mar-2024 23:24:41 JST Erik Jonker

     in reply to

     @johnpettigrew ...the paradox is that AI as an assistant is very useful if(!) you are already a good coder and can quickly check outcomes.

   • Embed this notice
     raganwald 🍓 (raganwald@social.bau-ha.us)'s status on Friday, 29-Mar-2024 00:11:22 JST raganwald 🍓

     in reply to

     • Cory Doctorow

     @johnpettigrew

     If you haven’t seen it, @pluralistic’s essay on the Generative AI Bubble makes a nice distinction between low- vs. high-value applications of AI, along with fault-tolerant vs. fault-intolerant applications of AI.

     Fault-intolerant applications are more expensive to deploy safely, and it is hard to replace fault-intolerant jobs: Supervising the AI is itself a full-time job.

     https://locusmag.com/2023/12/commentary-cory-doctorow-what-kind-of-bubble-is-ai/