do u think age matters

I'm 29 my boyfriend is 40 he's loving and caring and I'm happy with him I'm pregnant with his child and he wants to marry me ... but now the only thing I hate is that he's the father of my unborn baby; people tell me that he's too old enough to be in relationship with me like um too young for him I end up getting confused and now I I regret having unprotected sex with him Do u think age matters or not?? Like iv been trying for a baby for a long period now with my Ex bt iv got pregnant with this guy pls help me