it's really, really stupid
it's based on "AI is magic, yay! just turn it on, no problems, yay!":
"hackers abused a flaw in Meta's chatbot that allowed anyone to reset the password of any account that did not have two-factor authentication switched on. The bug tricked the chatbot into sending a verification code to an email address controlled by the hacker, rather than the account holder's email address on file, simply by asking it. The chatbot complied anyway"