Can we all start out in the womb as female?

Yes: Embryonic differentiation of the urogenital tract starts along a common pathway. It is the presence of male factors being expressed from the sry region of the y-chromosome that results in differentiation towards male genitalia. In the absence of many 'male factors' the default pathway is female. There are genetic conditions associated with being genetically male but phenotypically female.