It's just a matter of time. You aren't a familiar face. It usually takes 2-4 weeks to really transition.

I don't take younger than 11 months so this won't apply to you as much but I also do not constantly hold a child (never wear a child). I will sit on the floor with them, hold them on my knee a bit but I don't do it all day.