Emma Stamm
I'm working on an article about whether reasoning models’ outputs or “chains of thought” faithfully reflect their internal processes or not. I want to know how researchers evaluate "faithfulness." How can they be sure chains of thought aren't hallucinations?
Any resources you could point me towards would be helpful, including articles, people to talk to, etc.
(& this is yet another request where boosts would go a long way! 🙏 thank you thank you ) #genAI #LLM #LLMs
JP
@emma i don't have a specific link handy but @emilymbender has probably at some point spelled out exactly what "reasoning" means for these systems; my understanding is that it is just a pattern for building successive understanding-free prompts on the basis of an original human prompt, with the main benefit being that there's like a smidge more of a paper trail that humans can scrutinize rather than the complete black box of the classic prompt-response with tiny context window loop.
Prof. Emily M. Bender(she/her)
Hi Emma & JP, I don't think I have anything in writing, but indeed "chain of thought" is still just repeated responses to "what's a likely next word", based on additional training data that looks like "chain of thought".
A couple of relevant MAIHT3k episodes:
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.