Here is what I would do...
I personally think that it was to much and rather rude of the dad to make a snidey comment to their child at the door, rather than having the balls to ask you to your face if you'd mind. He knows he's pushing his look and asking more from you than he should. That being said, it is a request easily granted in a few minutes at very little cost. But, to prevent this particular family from taking advantage of you in the future due to your good nature, but also avoiding them thinking you are cold and caluss for not doing something which to them seems to be a small request, this is what I would do.
I would get this child up from nap just 5 minutes early and let them choose some different craft materials from your supplies and but them into a brown paper bag or zippy. I'd let them pick everything from tissue paper scraps, to buttons and die cuts etc I would include in the bag a folded piece of construction paper for the card. When Dad picks up, I would hand it to him and say, little Joey put this bag of craft supplies together to make a card for mums birthday. I would quite clearly say to Dad, I had all my activities planned for the day and thought it more appropriate and meaningful if you made the card with your child. Not only will this be a better experience for you, but it will also put you in the good books with your wife.
It's showing that you care about the child and family, but that you can not and will not change your days plans to accommodate an activity which is his role not yours. If he says he doesn't have time, I would even push it so far as to tell him it will only take 5 minutes with his child to make and that it will be worth the effort. I bet he'll never make some snide request like that again lol.