Babies know their mums via smell. Don't worry lah. They surely know who their mum is and who is not. My 2 mth old baby is also a preemie. He's mix feed and I will pass him to either my husband or my older children to look after him while I do my work od pump. He also cries alot. I even ask my elder son to look after him while I sleep (sleep deprive) 😅.
My girl is a preemie and I give her ebm since birth. Sometime I will feed, while other time my hubby will. She goes to ifc at 5.5 mo and she knows I’m the mummy! There’s many ways the baby will know you are the mum thru bonding. And of course our smell and voice