Is he going to sue you for one day's pay? I doubt it because that would be a lot more costly than the money he paid you. So if I were you I would be firm but very nice and professional at all times and tell him that according to your contract and the past offer to let them use the day and their refusal, that issue is behind you both.