Did either of the caregivers engage your baby at any point? I understand giving him time and space to get comfortable with these new surroundings, but if they just let him sit on a chair the whole visit, that seems rather indifferent.
Mind you, I am an absolute newbie to home daycare, but I studied to be a preschool teacher, am an experienced nanny and a mom of 3, and I wouldn't be able to just let a little guy like that sit all by himself for more than a few minutes.