I would put my children first for sure from what you have said and forget the job, plus it sounds tiring getting up that early and back later and not having time for your children or yourself by the sounds of it, I'd put the job behind me and look at it as a time of my life that is now behind me but now I have the time to teach my children again and time for them (which we don't get back again as you know). Ask them what they want. Sounds like it was exhausting for them too, I'd hate seeing my child so tired personally speaking and that in itself makes it harder for them with school work. It's a reason I am doing two night shifts a week max for now so son can be at childminders for bedtime until I am finished and collecting him at 7.30am next morning, and day care day time once my clients start coming, with before and after school care for ages 2-6 though ages 3-7 may be better.
Different if the job was on your doorstep I guess, as to the clients, be selective and do a contract that helps you filter out the trouble ones, sheesh I hope I don't have difficult clients that would put me off too..