I feel like we are getting so worried about our children playing ! We may be bubble wrapping soon !!! I get it ! I do , who hasn't seen someone run across and get hit by a swinger ! It's sad ! It hurts ! Most of the time , thankfully they live without severe injury . In a backyard and at the age you mentioned , I don't think I would pull the swings . I think it's a lesson we all learnt !
I am very much a hover mother with my last child ! Def !! And I treat my ducks pretty much the same , but I'm getting better at realizing kids fall , get hurt and heal !
So my advice would be keep the swing but as mentioned it's your dayhoke , do what makes you feel more comfortable :-)