I'm a cis gay man and I've never understood why gender is on public records at all. Totally agree with you.
If gender has never been on eg. public records we'd probably still have problems with gender in our stupid culture, but the arguments for m f or whatever on them would be mooted.
For the record I'm 100 percent behind the idea that everyone has absolute right to self determination and definition. And I say it out loud anywhere.