Ummm no!
Major red flags here!
I have been there with 1 family which is why I changed my business hours - it began to interfere with my own familys time and was WAY too long of a day for me.
I would send out a notice to them only and give a month, say that as of "xxx" your hours will be changing to 5:15pm. You know that he can accomodate that.
At least you know there is a light at the end of the tunnel. In the meantime if you are feeling like this arrangement is becoming too much give your notice. Family comes first!