5 months is a good shot!! If they are not showing any sign of improvement and it is stressing you out, I would terminate. I am like you and don't like to quit but I also plan on doing this forever so I don't want to burn out either.
Sounds like they have an inconsistent life (with "dads" coming and going) so you are probably one of the more stable relationships they have. I would give mom a time frame and say that if there isn't any improvement in their behaviour in that time, you will have to ask she find alternate care.