How to stop swaddling 3.5 mth old?

Hi mummies, my 3.5mth old is now able to loosen his swaddle and covers his face with it. We are trying to wean him off by releasing 1 hand but he uses it to rub his face and eyes and it wakes him up/ he can't fall asleep instead. If we totally don't swaddle his hands, he starts to rub his face the moment his hands touches his face and wakes up from it. Any advise/ sharing of experience will be appreciated. Thank you!