Yesterday I heard somebody who works for an "AI" company ask his coworker "Could you help me run this Python script? I tried just pasting it into ChatGPT, but that didn't work, so I'm not sure what to do next".

This is what "AI" companies are.

It's scary that this person would have taken whatever output that a glorified autocomplete returned to him, and he would have just treated it as correct because he didn't know any better to scrutinize it.

When these products change from being accidentally wrong because they're stupid, to being purposefully wrong because it helps their bottom line, nobody will notice.