Are there any disadvantages of not to letting baby have a pacifier? Like affect their brain development?

Not that I'm aware of. The only thing i can think of is you may not be able to soothe the child from crying. Other than that, It's more pros and cons. In fact, if baby is on pacifier, you will then have to think of ways for your child to wean off.