Frustrated With Some People In Lift

Just want to share my frustration please don't judge. Recently the lifts in malls are marked the 4 corners and it's generally required that only maximum 4 people can enter. I totally understand the reason and also respect it, however when we go out with baby in stroller and when the mall is crowded it's impossible to get in the lift. One time we were in a hurry and already waited more than 5 minutes, the lift arrived but already had 3 people so husband entered with stroller and I decided to take the escalator. I feel that should be acceptable because baby is small plus we're in the same household. But then a young guy in the lift told us off and even pushed my husband out?? We will never be able to use the lift if we must wait until there's only 2 people in the lift which is impossible during crowded times. Why can't they show some empathy and let us in? Also if people are able to use the escalator, why not use it instead of taking the space of those who can't? I feel so frustrated at that guy but also conflicted because he's technically following the rules, but doing so rigidly and without regarding for people's circumstances. What do mummies think, am I being entitled or am I right to be angry?