@theorytoe @tyler @unabomber As I understand it, this is a good explanation. We've known since before LLMs became a thing that we could use "fuzzy logic" to get close enough answers that actually surprised engineers with how well they worked. I recall reading something in the early 00s about the nascent concern of runaway issues related to this. Going from memory and grug understanding, paraphrased:
>It's fine to use tools for an educated guess, but you need to check your work and update it. What will happen when close approximations become relied upon and eventually become standards, compounded with people losing the knowledge needed to correctly solve the original question or verify its derivative conclusions?
This was 20 years ago, and it seems we're starting to see the beginning effects of this, but we won't know how bad it is for another decade+.