If your baby (babies!) was/were a little older, I would say "give it some more time".
But you have your hands full right now. If you think it's not a good situation, then it's probably not!!
Be honest with them. Apologize profusely, and give them at least a little time (if you can), to find someone new. If they get upset, don't let it bother you!! You are looking after the wellbeing of their kids as well as your own. I would hope that they can be understanding and appreciate your honesty. Good luck, and I hope you find just one more for your bunch!