I also only offer water - they will eventually drink it, just don't give up!
I would probably stop offering milk in the afternoons if you think it is discouraging them from drinking water.