Well, I have to say that is very childish behaviour on the part of the dad. I think I would first of all make sure to mention to him that it is totally normal for kids that age to have trouble with transitions (coming and going) and not at all personally directed at the dad. You can mention it in a light way...no big thing. As for the little comments, next time he makes one, hard as it is to do, I would say something. Ask him what he is getting at. Tell him you are sensing a feeling of dissatisfaction from him and would really like to know if something is bothering him so that you can discuss it. If he acts like a child and doesn't discuss things politely with you, then terminate.
As for being termed by parents, well it hasn't happened to me yet, but I can sure empathize with you...that would feel awful! I have terminated kids twice...the first was a combination of the child's behaviour and some disrespect from the parents...the second I just did and was because of the lack of progress in child's behaviour due to parents finding the behaviour "normal" and not stopping it at home.