GPT should get it wrong if you ask it enough. These things aren't intelligent. They can't get anything right or wrong because they're literally next word prediction algorithms. Not even next word, the next part of a word!
After all the transformers with their attention blocks, the end of the chain pick "should the next token be 'ng' 'ru' 'blu' a space or punctuation?
That's literally how they work. Anything they get right is purely by accident.