should I worry if my doctor says my baby is smaller then average ? But his healthy .

I wouldn’t. My baby also on the smaller side, came out at 2.6KG (40.3weeks) but all good.