Mother is responsible for the sex of the baby.
Father
Mother has XX pair of sex chromosomes while father has XY pair of sex chromosomes. Hence, mother always contributes an X chromosome, while father can either contribute an X or Y chromosome. If the father contributes an X then the baby will be a girl and if Y then the baby will be a boy. Hence, it is the father who is responsible for the sex of the baby.