I hear ya on what a peaceful week this last one has been!! I have had the same without 2 of the kids normally in my care full-time.
I think I would feel better giving them a chance to resolve some of these issues. If you are able, discuss the challenges you are seeing. Either that, or if you do feel like you have been giving in to him because you know that is how it is at home, I would toughen up the rules, lol. Even at his age, he will learn that he can get away with certain things at home that he cannot at daycare.
Ultimately though, if you don't feel like it's working out and you are dreading his presence... say goodbye. It may not be the right environment for him.