I would terminate. I have been in the same position when I serviced out of school clients which I don't do anymore. I have also had a client who left my daycare whose sister-in-law continued to come to my daycare after her sister-in-law left. I think people understand that circumstances change and these parents at the school have no idea, for the most part, what the situation was that led to the child not being at my daycare any longer. But the most important thing is your sanity so I wouldn't hesitate terminating if it is disrupting your life.
My unwritten rule for myself is that the parents must work with me to solve any problem that arises and if I can see that they are working with me I'll keep trying until I reach my limit. I hate to terminate families because that means interviewing and transitioning a newbie again which are my least favourite things. However, when we reach our limit we know it and it's so important to be happy and as stress free as possible because we have a really hard job to do every day and we work looooooong hours.