I really feel it does matter, both for comfort and safety (my LOs have fallen backwards/sideways and hit their heads against the mattress multiple times 😅). So I’m glad I always bought 4 inch mattresses.
I feel that it matters to offer enough support for baby. I bought a 3 inch mattress to add on to the one from the graco travel cot