It doesn't bother me at all. I look after children - babysitter, daycare provider - whatever. When I worked in an office I had no issues about being called a secretary either instead of administrative professional or whatever everyone else liked to be called. Each to their own.
That being said I do refer to myself as a daycare provider. I have also never had anyone call me a babysitter before so who knows how I would feel. I guess it all depends on the context of what was said.