I am a proponent for breast feeding as well.... but I do not think it would work out.
Unfortunately, that is the choice you make when going back to work, no matter what age the child is. I would hope that if you have a young baby (under 12 months) you continue to express milk, etc so that the breastfeeding can continue. A child 12 months+ can certainly make it through the day, and have mom nurse mornings and nights.
My primary concern would be the disruption to the children. Parents are absolutely welcome to swing by and get their kids at any time, but the thing that I like to make clear is: when your child sees you, it means home time in their little minds. Come on over, but you WILL be taking your child with you when you go.
Second issue with this is indeed the safety aspect. I do not allow parents to come and hang out with the kids, because other parents likely do not want their kids with strangers.