Does it matter if I choose a male or a female primary care physician?

It shouldn't. Most are comfortable with the fact that men and women share the same body of knowledge, and the same respect for patients and privacy. This is definitely true for primary care; your choice should for training, competency and 'bedside manner' The sex of the doctor should be irrelevant and in many structured care centers you may have no choice in the matter.
Comfort. All depends on your level of comfort. You should choose the physician with whom you can develop good, trusting patient-physician communication.