Should I be worried?

Hi! Just want to ask your opinion regarding sa situation namin lately. I am married to my husband for 5 years. We have a 11 months old baby girl and i'm currently 9 weeks pregnant with our baby boy. Team leader sa isang call center company husband ko, stay at home mom naman ako for almost 3 years na rin. Operations manager ako dati sa isang gaming company but my husband told me na to look after our child that time. Recently, may bagong lipat na office mate nya a few blocks away from our house. At first, wala lang sakin. Pero napapadalas na yung pagsasabay nila pagpasok at pag uwi. Husband told me na same shift lang sila kaya he's being polite na isabay on the way si girl. I asked my husband to stop na yung pagsabay sabay nila but he insisted na wala naman akong dapat ikapraning. Kung kayo nasa posisyon ko, what will you do? ?