Inhale Second Hand Smoke

I have on many occasions accidentally inhaled second hand smoke, such as walking pass someone smoking or someone smoking walking in front of me, but of course I would quickly move away, but can't avoid inhaling some. Is my baby at risk? Im very worried and concerned. Advise and opinions please?