What determines the sex of the baby, the man's sperm or the woman's egg?

Man's sperm. Woman's ova or eggs only carry the x sex determining chromosome amongst the 23 chromosomes. Whereas sperm carry either an x or a y chromosome amongst their 23 chromosomes. When joined an embryo's cells contain 46 chromosomes. Xx = female, & xy = male. A sperm containing an x chromosome after impregnation results in a baby girl and a y chromosome containing sperm will produce a boy.
The man's sperm. The sex of a baby is determined by the two sex chromosomes inherited from the parents. A baby will normally inherit one sex chromosome from the mother, and one from the father. A woman has two x chromosomes, so she can give either of her x chromosomes. The father can give either his x or y chromosome. A baby girl has 2 x chromosomes and a baby boy has 1 x and 1 y chromosome.
The father. Since a woman has two x chromosomes (46xx) and a man has an x and a y chromosome (46xy)- the woman can only contribute an x to the baby but the man can contribute an x or a y; therefore the man depends the sex of the child depending on which chromosome he contributes- x or y- if it is an x than the baby is a girl (46xx) and if it is a y, the baby is a boy (46xy). Hope this helps.