If I was in your place I would ask him to visit after the kids are gone