“Here’s the paradox: even while the language fluency fools us into imagining these chatbots are ‘persons’, we simultaneously place far more trust in their accuracy than we would with any real human. That’s because the fact that we still know it’s a computer activates another, more modern rule of thumb: that computer-generated information is accurate and trustworthy.”
It’s a brilliant essay; I highly recommend you take the time to read it.
https://buttondown.email/ninelives/archive/language-is-a-poor-heuristic-for-intelligence/