Are there any disadvantages of not to letting baby have a pacifier? Like affect their brain development?

I grew up using a pacifier as a toddler. Infact my maternal side even my aunts,uncles and cousins had pacifiers.....a few till they were 5 years old :-) Everyone has turned out normal ;-) Its just a notion or superstition that pacifiers affect the brain development. Maybe front teeth come a little forward but then these milk teeth do fall off ! Pacifiers do help them "stress free" and some peace to mommies. They give them up in due course of time. My son is 4 and needs it to sleep only but pushes it out once asleep or I pull it out.