In this Op-ed for Scientific American, Asmelash Teka & I discuss one of the many reasons the idea of replacing US federal workers with so-called generative AI systems should terrify us. We dive into the well defined task of automatic speech recognition (ASR), and describe why OpenAI’s Whisper, which has been integrated into ChatGPT, makes stuff up, or “hallucinates” as it’s called in the industry (bad nomenclature).