Totally agree with previous comments... I don't know that this is a termination issue, as yet. But I would be saying "This is our daily routine, and we will be having more children join it, soon. I cannot have your child arriving during naptime/quiet time, wide awake and ready to play. Quiet time is from this time to this time, please ensure that he arrives ready for it."
Don't feel "bad" about him arriving at your house just in time to go to bed, either. That is NOT your problem. If the parents don't like it, it's their deal.