They didn't just leverage white men's racism. They leveraged their own racism. They were white supremacists. Some white women are white supremacists. Susan B Anthony and Elizabeth Cady Stanton were two examples of white women who were white supremacists.
They argued against the *millions* of white men at the time who thought that both Black people and women should have the right to vote.