Public
- Public
- Network
- Groups
- Featured
- Popular
- People

Conversation

Notices

Embed this notice
Prof. Emily M. Bender(she/her) (emilymbender@dair-community.social)'s status on Friday, 15-Nov-2024 08:58:43 JST Prof. Emily M. Bender(she/her)
- Alex Hanna
Mystery AI Hype Theater Episode 44: OpenAI's Ridiculous 'Reasoning'
https://www.buzzsprout.com/2126417/episodes/16095327-episode-44-openai-s-ridiculous-reasoning
In which @alex and I bring a little actual science to the party to cut through the hype.
Thx to Christie Taylor for production!
In conversation Friday, 15-Nov-2024 08:58:43 JST from dair-community.social permalink
Attachments
1. Audiogram with MAIHT3k logo, ep 44 title and subtitle “How o1’s ‘complex chains of thought’ are assessed with meaningless metrics -- and erase what thinking really is” EMILY M. BENDER: The funniest thing--so this is where they're testing it against the International Olympiad in Informatics problems, I guess. So they're doing some kind of supervised learning effectively over this specific task. And then they say, "This model competed in the 2024 IOI under the same conditions as the human contestants." But, "It had 10 hours to solve six challenging algorithmic problems and was allowed 50 submissions per problem." But the thing that cracked me up about this is, "With a relaxed submission constraint, we found that model performance improved significantly. When allowed 10,000 submissions per problem, the Model achieved a score above the gold medal threshold even without any test time selection strategy." ALEX HANNA: I too would like to have 10,000 submissions per every exam I am taking. EMILY M. BENDER: I would never want to be the one grading that exam. So SJayLett says "'Under the same conditions.' It had an itchy nose?" ALEX HANNA: Yeah, this whole thing is a bit bizarre. And it's sort of, the analogy of chain of thought to sort of steps of human reasoning is, you know, a further kind of devaluing of what it means for humans to reason and how humans do it. [Finished in next post]
2. Domain not in remote thumbnail source whitelist: storage.buzzsprout.com
  
  Episode 44: OpenAI's Ridiculous 'Reasoning' - Mystery AI Hype Theater 3000
  
  The company behind ChatGPT is back with bombastic claim that their new o1 model is capable of so-called "complex reasoning." Ever-faithful, Alex and Emily tear it apart. Plus the flaws in a tech publication's new 'AI hype index,' and some palette-...
- Timnit Gebru (she/her). repeated this.
- Embed this notice
  Prof. Emily M. Bender(she/her) (emilymbender@dair-community.social)'s status on Friday, 15-Nov-2024 08:58:41 JST Prof. Emily M. Bender(she/her)
  in reply to
  
  Also available as video on PeerTube:
  https://peertube.dair-institute.org/w/gAAnkju7qjfrWjG9NZVy2L
  
  In conversation Friday, 15-Nov-2024 08:58:41 JST permalink
- Embed this notice
  Prof. Emily M. Bender(she/her) (emilymbender@dair-community.social)'s status on Friday, 15-Nov-2024 08:58:42 JST Prof. Emily M. Bender(she/her)
  in reply to
  
  End of audiogram alt text:
  EMILY M. BENDER: SJayLett adds, "I suspect it emphasizes how the people behind this don't really believe the mind is embodied in any meaningful way," which I think is very true.
  
  In conversation Friday, 15-Nov-2024 08:58:42 JST permalink
  
  Timnit Gebru (she/her)., Jure Repinc :linux: :kde: and Michał "rysiek" Woźniak · 🇺🇦 repeated this.

Public

Conversation

Notices

Feeds