I think you just need to be more firm with him. Whether or not he naps, you both need that quiet time. He should have to amuse himself with whichever books/puzzles/individual games are made available to him....and/or a bit of TV time. I tell them that I'm recharging (including my ears), so I'm off limits. Luckily right now I have ALL nappers...even my 4yo dcb.