The challenge with attempting to only deal with mom is that, chances are, that won't work for them! Sometimes parents absolutely have to split up the kid pick up arrangement. And if you refuse to deal with dad, I think it likely that they will be gone.
I would give a firm and final warning and remind them (although it shouldn't be necessary!!) of how you expect to be treated. And then no more "friends" with dad, strictly business, as you mentioned. If he feels like he is on that "level" with you, he will feel more comfortable taking out his crappy day on YOU. Ugh, what an uncomfortable arrangement for you. Perhaps you will be able to outright discuss this with both mom and dad present (I don't think you should just talk to mom, messages get mixed up that way sometimes) and save this. Otherwise, I would say goodbye, even if you love the majority of the family. It's not worth worrying if today is the day he's going to yell at you or your daughter or staff again.