Do you think he is watching you for some reason? Or bothered by the noise of the kids? but he is at home with his own kids?....maybe you can acknowledge his watching somehow. Can you come in contact with him, to talk?