Conversation

Notices

1. Embed this notice
   Janelle Shane (janellecshane@wandering.shop)'s status on Thursday, 30-Mar-2023 00:23:13 JST Janelle Shane

   Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.
   https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
   #gpt4 #AIHype #ThisIsWhyWeDontTestOnTheTrainingData

   • Embed this notice
     Thomas 🔭🕹️ (thomasfuchs@hachyderm.io)'s status on Thursday, 30-Mar-2023 00:23:10 JST Thomas 🔭🕹️

     in reply to

     @janellecshane
     Not sure I get what's surprising about a result like this.

     LLMs are just word salad generators coming up with text patterns that reflect their training data. It's literally the only thing they do. ?

   • Embed this notice
     Thomas 🔭🕹️ (thomasfuchs@hachyderm.io)'s status on Thursday, 30-Mar-2023 02:34:33 JST Thomas 🔭🕹️

     in reply to

     • AgreeableLandscape

     @AgreeableLandscape @janellecshane LLMs aren’t actually good at these things (that's why it gets 0 points at the more recent questionnaire), they're only good at generating text that they saw before in the same context.

     Other (non-AI) algorithms are actually designed to work in any situation (like in the chess example).

   • Embed this notice
     AgreeableLandscape (agreeablelandscape@mastodon.social)'s status on Thursday, 30-Mar-2023 02:34:34 JST AgreeableLandscape

     in reply to

     • Thomas 🔭🕹️

     @thomasfuchs @janellecshane Seriously tho how is this even news? IBM had their Watson robot play Jeopardy over a decade ago. WolframAlpha can also solve tons of high school and college tests as long as they're not too cryptically worded. Of fucking course computers are good at doing knowledge and logic based questions! They're *really* overselling chatGPT here by claiming this is some new innovation.