What you did in your hypothetical is to point to a (theoretical and unspecified) safety concern NOT related to sex, and complain that sex category didn't remove it. You're applying an impossible standard: you demand that when sex category is established, then no safety concern must ever present for any reason ever, even theoretically.
We don't expect that standard from any other measure.
Yes, sex testing would mostly affect female sport, that's true.
No, for nearly all athletes, except only for the very rare cases where more diagnostics after screening is required for health reasons anyway, it's a cheek swab. This is not invasive.