No matter HOW MUCH fun your house is, it's not his house. And 3 is exactly the age that kids start catching on to that, and the idea that perhaps what they say will influence you/mom/dad,etc.
I wouldn't worry about it, or take it personally, AT ALL!! As long as he still seems like he is having fun, he probably is! If it starts to become a regular thing he voices to you, I would let mom and dad know that this is what he is saying BUT he is still doing well, and that you just wanted to let them know.