As long as you are doing stuff that you enjoy than that is fantastic!! I'd also classify some of that as downtime. When I do my artsy projects, that's my downtime, my time to destress!! It's when we get caught up in doing things we feel obligated to but don't want to do, that's trouble!! Saying no is okay!!
I've had one family for sure leave for that same reason! She only got 2 weeks vacation and unless she wanted to take it at same time, I can see how annoying that would be. It's why I stress so much during interviews to have a plan for my holidays in place. I refuse to feel guilty (well more than I already do! Lol) for taking some me time.....just like many of them do!