It sounds like she would be coming right before nap time.
That means, that potentially the baby would start having a meltdown/cryfest when mom leaves and then refuse to nap for you.
I think you made the right call.
I can see mom's side too but the potential for disruption and being unable to settle the baby again would be too risky for me.