It's due to the amount of pumps. Lesser pump = lesser demand. Thus your boobs will lessen the milk supply. What you can do is squeeze in power pump sessions. IT will help to stimulate more milk.
the more you pump/latch the more supply you will have. If i were you I'll hold everything off and pump. nothing is more important then milk for my lo. that's just me.