@futurebird @dalias
I'm sure you saw the bit about Dawkins spending time with an LLM and announcing that he thinks it's actual intelligence somehow. To me, that illustrates the danger of approaching it as something that resembles the simulation of a human mind.
LLMs ARE the distillation of the human mind in a way. A rotgut, prohibition era, bathtub-gin type of distillation but there you have it. I think it's fundamentally more likely that we would decide non-intelligent actors are intelligent because they ape that model, when an actual generated intellect would be.... whatever it is.
We made gods in our image, too. We tend to understand our world via human metaphor.
There's a scenario for AI where no one recognizes what it is until it cannot be stopped, because people disregarded it at first as it didn't fit the image they had for AI.
I myself have the hopelessly subjective notion that if artificial general intelligence arises it will be under no particular urge to clue us in that it exists :)