Who is carrying your baby out of the room? Since you’re the mother, you can be more firm about wanting to spend time with baby, dear. Allocate time for the rest of your family to play with baby. You decide that. But of course, you would also expect to have to be more independent in taking care of baby (feeds, bath, etc) because your family may feel that they’re helping you. Talk it out nicely and let them know how you feel:)
Could it be they want to let you to rest more so that you are able to handle your baby at night? If isn't, you may want to tell them nicely to let you spend some time with your baby during the day. If they are your in-law, then let your husband be the speaker.
Try talking to your family members if you are very bothered by their behaviour. Maybe they meant well and just want you to rest more in the day so that you can do the night duty.