Baby keep scratching face

My 1.5mth LO has been scratching his face very badly, especially when he's tired or hungry. We've cut his nails ,but all the swiping he keeps doing still left his face all red and peeling. I don't think I can keep having him wear mittens already. Any idea what to do? During the night, we tried swaddling him to stop his hands from scratching,but when we unswaddled him for feedings, he scratches and swipes very furiously, and gets very upset from being woken. Before we swaddled him, there wasn't this issue. He would drink his milk and go back to sleep peacefully.