My newborn doesn't let me carry him

Just FYI, I am his mom. I breast feed him at night and bottle feed him in the day time. I change his diapers, play with him, do development with him and put him to sleep. But I want to be like other moms and rock my baby to sleep, carry my baby and talk to him etc. But everytime I carry him, or try to, he screams murder and struggle like mad. I have no choice but to put him down and he will stop screaming but continue crying. Someone has to take over to calm him down. I try every possible way of carrying him to no avail. The only way possible for me to calm him is to put him to my breast - breastfeed him. Everyone else has no issues carrying him - his grandma, grandpa, dad, even a stranger - only me. This makes me very sad. He doesn't even smile much or talk with me. Everytime I try to talk to him he stares at me blankly. His dad walks past and he saw him and he starts smiling and babbling. His grandma and grandpa too. Only me. I feel like my baby hates me.