Who are the 'wild men'?

[Inziladun]

Originally Posted by Elmo

I'm sure the Dunlendings migration happened in the Dark Years of the Second Age, where they moved North to escape Sauron's shadow and so long pre-dating the establishment of Arnor.

Alien too, or only remotely akin, was the language of the Dunlendings. These were a remnant of the peoples that had dwelt in vales of the White Mountains in ages past. The Dead Men of Dunharrow were of their kin. But in the Dark Years others had removed to the southern dales of the Misty Mountains; and thence some had passed into the empty lands as far north as the Barrow-downs. From them came the Men of Bree; but long before these had had become subjects of the North Kingdomof Arnor and had taken up the Westron tongue.

ROTK Appendix F

It doesn't give the reason for the move north, but you could be right. Point is though, that those migrant Dunlendings didn't harbor enough ill will toward the Dûnedain to induce them to resist Arnor. The Bree-men, at any rate, had no qualms with being subjects of the Northern Kingdom.