Where I live is probably something to do with it, and I'm sure white male privilege is a non-trivial part too, but I've been fortunate so far that nobody's hassled me for wearing a mask.
I can tell myself I'd have all kinds of clever responses if someone did, but I have no idea how I'd react in reality.