I love this observation, and all the ways it opens up of still writing trans characters.
Although it doesn't necessarily hold true with nonbinary characters, because they by definition don't fit in unless the entire world accepts genders other than man and woman.
I'm in that light-skinned, somewhat buff, slim-enough-to-have-privilege category of androgynous presenting in meatspace that I get to experience pretty regular gender confusion as people attempt to work out whether I'm a very masculine woman or a particularly feminine man, and that would equally apply in a fantasy universe where I could magic my body into my most desired form. More so, even.