Chromosomal sex is determined at the time of fertilization; a chromosome from the sperm cell, either X or Y, fuses with the X chromosome in the egg cell. ... Over the next five weeks, the fetus begins producing hormones that cause its sex organs to grow into either male or female organs.