So, it's not true to say that all of us start out as female. It also might not be true to say that female is the default developmental pathway in absence of certain genes and hormones expressing.
So why do so many people believe that we all start out life as female? There's two possible explanations: a sexist one, and a not sexist one!