Yup, I have absolutely met kids that I didn't "click" with.
As Reggio said; they are people, just smaller! And some people just don't mesh well.
If you don't feel like the child is suffering for it, then I wouldn't worry. It may even get better.
I had one little boy like this. We got off to a rough start, and I just couldn't get past it. I was actually relieved when I had to say goodbye.