Dare Obasanjo
OpenAI published a paper showing that LLMs hallucinate for two primary reasons: first, due to statistical errors during training, and second, because LLM evaluations reward confident, even if incorrect, guesses but penalize “I don’t know”. So LLMs are trained to always guess when they aren’t sure.
The paper proposes that to fix this, we must modify LLM evaluations to properly reward models that acknowledge when they don't know the answer.
draeath
@carnage4life "properly reward models that acknowledge when they don't know the answer"
Wouldn't that mean rewarding every output? /hides
Rich Felker
@wcbdata @carnage4life I mean for answers making mathematical claims, you could evaluate them. This would fix some of the stupidest stuff like arguing about number of occurrences of a letter in a word or values of mathematical expressions. But it won't fix the larger fundamental problems.
Bill, organizer of stuff
@carnage4life Interesting statement, but meaningless. LLMs fundamentally cannot work that way. To "know the truth," you need a deterministic, rules-based system - very different from (and wildly more complex and expensive than) an LLM. There was an excellent mathematical proof of this a few years ago, but I can't be bothered to go find it at this hour. To oversimplify a bit, it's like saying that to make fire safe we need it to stop when it's burning something.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.