Public
- Public
- Network
- Groups
- Featured
- Popular
- People

Conversation

Notices

Embed this notice
Jan Wildeboer 😷:krulorange: (jwildeboer@social.wildeboer.net)'s status on Saturday, 07-Jun-2025 22:56:38 JST Jan Wildeboer 😷:krulorange:

On the limits of LLMs (Large Language models) and LRMs (Large Reasoning Models). The TL;DR: "Our findings reveal fundamental limitations in current models: despite sophisticated self-reflection mechanisms, these models fail to develop generalizable reasoning capabilities beyond certain complexity thresholds." Meaning: accuracy collapse.
Interesting paper from Apple. https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf
#AI #LLM #LRM
In conversation about 7 months ago from social.wildeboer.net permalink
Attachments
1. • We question the current evaluation paradigm of LRMs on established math benchmarks and design a controlled experimental testbed by leveraging algorithmic puzzle environments that enable controllable experimentation with respect to problem complexity. • We show that state-of-the-art LRMs (e.g., o3-mini, DeepSeek-R1, Claude-3.7-Sonnet-Thinking) still fail to develop generalizable problem-solving capabilities, with accuracy ultimately collapsing to zero beyond certain complexities across different environments. • We find that there exists a scaling limit in the LRMs’ reasoning effort with respect to problem complexity, evidenced by the counterintuitive decreasing trend in the thinking tokens after a complexity point. • We question the current evaluation paradigm based on final accuracy and extend our evaluation to intermediate solutions of thinking traces with the help of deterministic puzzle simulators. Our analysis reveals that as problem complexity increases, correct solutions systematically emerge at later positions in thinking compared to incorrect ones, providing quantitative insights into the self-correction mechanisms within LRMs. • We uncover surprising limitations in LRMs’ ability to perform exact computation, including their failure to benefit from explicit algorithms and their inconsistent reasoning across puzzle types.
  https://cdn.masto.host/socialwildeboernet/media_attachments/files/114/641/447/304/198/545/original/4adb1e249fc6bc65.png
2. Untitled attachment
- Tim Chambers and Rich Felker repeated this.
- Embed this notice
  Jan Wildeboer 😷:krulorange: (jwildeboer@social.wildeboer.net)'s status on Saturday, 07-Jun-2025 22:57:00 JST Jan Wildeboer 😷:krulorange:
  in reply to
  
  Further proof that "AI" behaves just like a sealioning reply guy ;) Even if told what the working solution is, it simply ignores it.
  "We uncover surprising limitations in LRMs’ ability to perform exact computation, including their failure to benefit from explicit algorithms and their inconsistent reasoning across puzzle types."
  Idem, Page 3
  
  In conversation about 7 months ago permalink

Public

Conversation

Notices

Feeds