is it a mutual annoy thing or is one of them more likely to be the one doing the pestering first - maybe minor but just playing too close in the other child's space because they know the child doesn't like it is enough. The fact they play fine when you are there suggests they may have learned that the behaviour gets your attention away from the little ones. Have you tried not taking sides in a dispute and in effect punishing both of them? Time out is a concept not a reality thing as there is no right or wrong way to do timeout as in doesn't come with a manual so if they can't play well without being with you then I guess they need to be with you - that means sitting there cross - legged a few feet from where you are interacting with the little ones. They do not get to participate in any way. After a few days of "timeout" doing nothing they might get the hint. Then ask if they would like to go do - and give them only one choice like puzzles or books something that is an alone activity but in the same space. Repeat till they can manage to stay and play.

Since your child is one of the oldest and one of the youngest be very aware of his behaviour. Is he jelous of the baby as in is the fighting worse if you move from them to the baby. He may very well be the instigator as a way of getting attention away from the baby. If that is true then again he can come and sit while you play with the baby if he wants but doesn't get to participate - he had his turn for your attention.