Not sure what you mean by "right down town." You want to be close to down town but not right downtown (meaning Dundas and Yonge) because although people work there, less families live downtown, more singles, older empty nesters. I would think you should look at the areas I previously mentioned above as they are in neighbourhoods with young families and you need to have some money (middle-high income families) to live there too. Plus they are close to downtown.