Like the others have said, do not feel guilty about having a vacation with your family. This is important time for you to rejuvenate yourself and enjoy your family. Anyone who begrudges that is quite simply selfish and self absorbed.
I just handed out my yearly vacation dates. I am taking 15 days this year. I don't get paid for this time off and my clients have plenty of time to find alternate care or vacation the same time. That is their problem not mine as I have stressed they must have back up care.
Hard to do I know, but ignore their reaction as they are disrespecting you.