The word man originally was gender-neutral. Wer was used for male-gendered men and Wif for female. Wif persisted as "wife" and wifman->woman. The wer was dropped in the 10th century or so and man just came to take over.
Which is fun because that wer is the same as werewolf, so there also used to be weremen