Can you try to 'force' him to be held? but I mean it in a good way LOL
My brother hated to be held or hugged, even as a baby! So at times my mom would have to actually hold him and MAKE him calm down.

I'm at a lost....sorry hun. All I can say is I hope your day is better than the last.