We sometimes say "Someone being gay or trans doesn't effect your life at all." Things along those lines.
Thing is, this isn't totally true. There *is* an impact.
A good impact. A healthy one. Trying to enforce ideas of gender and sexuality that don't align with the complex way that we really *are* has an impact on everyone.
If being non-binary is "wrong" then not being a perfect match for your gender is wrong.
Maybe some people are scared of who they would be without the fear.