Twiddling actually helps your baby calm down and focus more on nursing thus helping the feedings run smoothly and baby gets to drink more. It also serves as a self-soothing mechanism so baby might twiddle more when they are tired or fussy.
If it bothers you so much, you should stay calm, look baby in the eye and gently say no. Pull baby away your chest each time this happens. In the long run, he will associate twiddling with negative behaviour and stop. Alternatively, keep the free nipple covered. If he can't see it, perhaps he will not be inclined to twiddle.
Read more