I have the following policy:

Custody Arrangements
It is my recommendation that all separated/divorced parents provide a copy of the custody arrangement. I would be UNABLE TO ENFORCE any custodial parents requests WITHOUT legal documentation.

This copy should include the following.

TITLE PAGE (this states who is involved),
CUSTODY PORTION (this clarifies the agreed upon arrangements), and
THE JUDGES SEAL & SIGNATURE (this confirms it has been reviewed by a court of law).

Otherwise either parent has the right to pick up. As for the grandparent, unless you have it in writing from the parent that signed your contract that they can pick up. Sorry, they can not pick up the kid.

But yes I agree with your husband. This is trouble either have them come up with some kind of arrangement in writing or terminate them.