@emilymbender The obvious counter to this is that LLM are not strictly given _text_ but also _communications_ and in particular through RLHF _actual communications and feedback on its "utterances"_.
I agree with the basic point that LLM are far more appearance than substance, and that people are very good at finding meaning where there is none, but there are pathways open for actual LLM as they exist or might exist that are not present in your example.