I had to terminate my neighbour's 4yrold boy for severe behaviour issues. It was very awkward in the beginning, but it got better over time - especially once he was diagnosed with a sensory disorder, and I think they kind of woke up and realized that it wasn't just me, that there was an actual, real issue that made home care an unsafe option for him. If/once you do it, give them space to get over it, but be friendly and polite when you see them - hopefully they'll eventually realize that you were in the right, and respect you for doing the awkward but necessary thing.