I wear yoga pants and sweat shirts every day. I don't wear makeup, either. This does not affect my ability to care for children in any way, shape or form, and anyone who thought that it did would be getting their notice.
Who cares if you had your bathrobe on? How dare she comment??? If her children were unhappy with the way you care for her grandchild, he likely wouldn't be in your care. One has nothing to do with the other, and I think she was incredibly rude to question you. It's one thing to think something. It's quite another to question someone's attire when it's no business of yours.