Yikes ~ this will be hard but you have been thinking about for a while and you know it's the right thing to do ( for your sanity )
I think maybe if you talk to the parents like the other ladies were saying to give them more of a head's up that this might be coming ...if it will make it easier on you. I've never been in this position and easy for me to say just end it but it's causing you enough stress that your husband is now telling you to end it. You can always put it on you and not so much their little guy saying that as much as you have tried you might not be the right provider for him.
Good luck !!