We used to live in townhouses and had that same problem. I didn't do daycare at the time, but as soon as I was out with my crew, all the other kids wanted to come out, too. Which would be fine except that their parents assumed that I wanted to watch their kids.
It takes a village and all that, I get it. But when 99% of the time I was the only parent out with the kids (who were all young and preschool), it gets old pretty quick. Especially when these kids would not listen and were little trouble makers. My solution was to actively ignore the parents giving 'instructions' and to only step in when someone either got hurt or was about to. Or take my kids elsewhere to play.
Or move, which is what we ended up doing. Not for that reason although I sure don't miss it.
Hope your backyard gets organized soon!