Trans men are men. Trans women are women. Non-binary people are...well, whatever they say they are... that's a mixed bag of individual identities, to say the least.
People who want to restrict trans rights want to restrict human rights. They want to set the boundaries of what it means to be human and force everyone to obey.
Transphobia is a form of proto-fascism.