[The Might]

No, you are quite right.
One can notice how low Gondor's population was (as a result of all the wars, plagues and all that) since that is why they gave up Calenardhon in the first place, and since also Minas Tirith still had room for many more during the War of the Ring.

And indeed the Rohirrim were brought close to extinction by the Long Winter.

Interesting would be to know how many Hobbits lived in the Shire, they were also severely affected by calamities like the Fell Winter, but they little to do with wars, and we know some had quite many children.