Is it ok to let a 2 month old baby to drink a sip of water? Maybe aboout 5 to 10ml?

yes it maybe heaty. alternatively it could be possible that you may have to switch milk. some babies get very heaty and constipated due to the milk. it's more of trail and error. but water definitely not for a 2 month old baby.