[The Might]

So what?
Tolkien said that if we were to consider M-e really existed then it would be about 6000 years between the Fall of Barad-dur and modern times.

So that means 4000 BC.
Now I dunno how many people lived in Europe back then during the Bronze Age or whatever age but I doubt there were many.

So that it makes sense that after much reproduction we now have all these millions that all originate from those first 1 - 2 millions. Of course in this case the Elves (who anyway were only very few) and Dwarves (same goes for them) don't count.