I know you're not supposed to stick a cotton bud into baby's ear, but I see some dirt there. Should I just let it be?

Yes. Let it be. It's perfectly fine to leave baby's earwax unattended as it will not cause infection nor affect their hearing. Do not insert anything into baby's ear canel as it may damage the eardrum. Only clean baby's ear external with wash cloth while bathing. http://www.parents.com/advice/babies/health/how-should-i-clean-my-babys-ears/