Weaning off pacifier

I started giving my boy pacifier quite early, at about 3weeks old. I was stress about the crying and felt very lost, hence i resort to giving pacifier to “calm him down”. Though my LO is nt very dependent on the pacifier but at times I can see that he wants something to suck (even though he is not crying for it). Sometimes he needs it to sleep, but sometimes he is ok without. How can I wean him off the pacifier and not affect his sleep.