Tim Chambers
Question for the Fediverse hive mind. Is there any evidence that the #AI models hallucination rates are getting ANY better over time?
I'm wondering if a 5 to 15% hallucination rate may just be the nature of the best with LLM's and an unsolvable problem.
Jörn Franke
@tchambers there is no specific hallucination rate of a LLM. The errors of LLM change depending on the context and what you ask. It is impossible to measure a specific rate - it is a nonsense measure as it always depends on the context.
Tim Chambers
@jornfranke I saw this effort that at least tried to create a means to do so: https://huggingface.co/blog/leaderboard-hallucinations
Tim Chambers
@jornfranke Maybe that technique with the BIG assumption they’re open about the training data - would be the best way to test this?
Jörn Franke
@tchambers well this is based on datasets the LLMs are actually trained on.... It is unrealistic that users will ask the same questions. Then, depending on the context with prompt that can be completely different. Also the safety guards put by some LLMs may obfuscate the results for "publicly known datasets" in either direction. The leatherboard has the same issues as LLM: Nobody can verify that the leatherboard delivers meaningul results.
Tim Chambers
@erlend great find, thanks!
Erlend Sogge Heggen
@tchambers there’s much more evidence to the contrary:
https://garymarcus.substack.com/p/evidence-that-llms-are-reaching-a
Tim Chambers
Looking through this too: https://arxiv.org/abs/2401.01313
Tim Chambers
And looking over this: https://arxiv.org/html/2406.04175v1 #AI
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.