Embed Notice

Another fun thing is https://en.wikipedia.org/wiki/Model_collapse

Feeding LLM output into LLM training is like sticking a microphone into a speaker to get a shrieking noise.

LLM output is POISON to LLM training data, the compression math collapses with fairly small amounts of contamination. Finding clean sources of data was already a problem back in 2023.

Another reason despite all the frantic scraping of the internet over the past few years, it keeps coming back to piracy:

https://bsky.app/profile/lynnemthomas.com/post/3moksjjsyxk2a