Sometimes, when we feel badly about something, our vision regarding the situation is a little skewed. I feel the same way about volunteering at my children's school, or being able to pick the kids up and pop over to the mall after school, etc. Seems like everywhere I look are smiling, happy Stay-at-homes with perfect hair and nails, and HOURS of free time on their hands. LoL
In reality, most parents are probably in the same boat as you, in that they cannot take the time off work easily, either. I do find it difficult to book doctor/dentist appts, because I can't just take an long lunch break or leave a little early.
Currently, my hubby is working from home, which makes things approximately one-million-times easier for me. He can pop out to the store if I am missing something for lunch or craft, or he can stay home while I run an errand or appointment during nap. Previous to this, though, I would find a back-up (usually my sis-in-law, or in a pinch, my mom) for an hour and a half at nap/quiet time. I would always ensure that the parents were aware/okay with this.