How long have you been using it? Maybe you are uncomfortable and have yet to get the hang of it. Baby wearing we need to adjust till we feel comfortable ourselves.
You could try shifting the waist strap up to see if it helps. Otherwise you may want to consider trying carriers with a softer waistband.