It's okay if the Dad is upset. Most people get upset when they are told no. Bless his upsetedness and do what is best to make sure that you, your children, and the other kids aren't upset. At least, with this decision, you just have one person in the mix of people you are considering upset. That's a good thing.