@Miniver One of the primary cognitive dysfunctions that occurs in psychosis is a failure of what is called "reality testing". The result is that the person cannot tell delusions from reality, and this makes finding a path toward mental health much more challenging.
Generative AI is at its very core psychotic, because it fails so spectacularly at reality testing. Which wouldn't be so bad, except as you note, people treat it as authoritative. A psychotic authority.