I ask my kids not to use "oh my God", use "gosh" or "goodness" instead. We also don't say "stupid" or "dumb". I have a 5 year old that we call the language police, because if we let anything slip out, she is ON IT. LoL
I know it doesn't seem like a big deal to most, but they will pick it up soon enough away from home. It just sounds wrong from a little 2 yr old, you know?