[The Might]

But all those forests were cut down by the Numenoreans in their quest for ever more timber to build their ships.

This means that the early Dunlendings did indeed live near the Numenoreans in times when Elves still wandered through those woods, but that their descendants probably saw no more Elves passing nearby since there was little reason for them.
In this case again, my idea would be that myths preserved the memory of Elves of Hollin or the Grey Havens wandering through the great woods of Enedwaith.