Although each egg cell produced by a woman carries a single X chromosome, the sperm cells produced by a man carry either an X or a Y. This means, in very simple terms, that the sperm cell determines a baby's sex.