Should I be worried that my son's the smallest in his class? He's 5

There is nothing to be worried about. They will grow and next thing you know, they shoot up real tall and double in size. As long they are happy and healthy it's very important!