Breast shrink

Hi mummies, I am at week 24 and noticed that my breast shrink. During 1st trimester my breast grow. Should I be worry? Thanks