I swing towards "often".
I don't think you ever have an obligation to leave words or images you regret anywhere on the Internet. Especially if they hurt someone, there's no point in hurting more people, or hurting the same people again.
If there's enough of a discussion around the post that you feel like it'd be unfair to remove it, I think it's perfectly OK to replace it with an apology ("I said something I wish I hadn't here") or just "deleted".
If it's a privacy issue, definitely delete.