I think if you are okay with the friends popping in then so be it, but that doesn't mean you shouldn't say anything to them when their child is physically harming one of your daycare kids. If you don't discipline them, then it sends the wrong message to your daycare kids and suddenly all your hard work is undone as they start to test the boundaries with you. For goodness sakes don't be passive about it like you were, say something to your friend or just outright confront the misbehaving child if their mother isn't bothering to do it.
The other thing to consider is how happy your daycare parents would be about someone they do not know spending time with their child. Most of my daycare families over the years have asked who else will be spending time with their child and I think it is a valid concern if it is someone on a regular basis. I understand this is outside of your control when they show up unannounced but I would either be saying no to visitors or informing the parents of what's going on. If a parent shows up unannounced and sees another adult in the company of their child, they may start to ask questions and wonder what else is going on that they don't know about.