You can't let someone else lower your self-esteem, because that's what it is - self-esteem. You need to first love yourself before you have anybody else love you.