@ptesarik @ben @pavel @vbabka the big problem is that people are very very bad at picking up on the kind of errors that an algorithm can generate.

We all implicitly assume errors are 'human shaped' i.e. the kind of errors a human would make.

An LLM can have a very good grasp of the syntax but then interpolates results in effect randomly as the missing component is a dynamic understanding of the system.

As a result, they can introduce very very subtle bugs that'll still compile/run etc.

People are also incredibly bad at assessing how much cost this incurs in practice.

Having something that can generate such errors for only trivial tasks strikes me as being worse than having nothing at all.

And the ongoing 'emperor's new clothes' issues with LLMs is this issue is insoluble. Hallucination is an unavoidable part of how they work.

The whole machinery of the thing is trying to infer patterns from a dataset, so at a fundamental level it's broken by design.

That's before we get on to the fact it's needs human input to work (you start putting LLM generated input in it completely collapses), so the whole thing couldn't work anyway on any long term scale.

That's before we get on to the fact it steals software and ignores license, the carbon costs and monetary costs of compute, and a myriad of other problems...

The whole problem with all this is it's a very very convincing magic trick and works so well that people are blinded to its flaws.

See https://en.wikipedia.org/wiki/ELIZA_effect?useskin=vector