Not allowing husband to bring infant out alone during weekende

I am a full time working mum while my husband is a SAHD. During weekdays I am busy with work and weekends are the only time I bond with my baby. My baby is on breastmilk and we direct latch during weekends as well. Due to Covid-19 restrictions, my husband's friends organised gathering but the max no of visitors is 5. My husband wanted to bring baby alone for the outing (without me) due to the restrictions in number of visitors. I refused and my husband got angry. Is it reasonably for me to not allow him to bring baby alone? I just want to spend more time during weekends with my baby and let him latch. Did I do anything wrong?