Icky. In an ideal world I'd say terminate! But since we can't always do that...
Like Lou said, don't respond to anything not related to his kid. And if he still has the guts to say something, text or at the door, the next time I saw him I'd hand him a sealed envelope with a notice telling him his behavior has been consistently inappropriate and makes you uncomfortable... If it continues you will request mom do the pick up and drop offs (because that would be a fun conversation for him), or termination.
It IS sexual harassment. Just because you aren't in an office doesn't mean he's allowed to treat you like a piece of meat.