[Zigûr]

I suppose it's not really explained in the films, is it?

In the books, the 'wild men' who serve Saruman are the Dunlendings, Men who inhabited Dunland, the region to the North of Rohan. They did live in "Rohan", or Calenardhon as it was then known, before the Éothéod (as the Men of Rohan were then known) came, but they effectively seem to have lived there as "squatters": although the Men of Gondor didn't really live there, Calenardhon/Rohan was technically the property of the Kingdom of Gondor and the wild men seem to have lived there without any real permission.

At the end of the war against the Balcoth (a group of Easterlings), the Éothéod were given Calenardhon/Rohan by Cirion, Steward of Gondor, in return for their aid, and they founded the Riddermark there, which is to say the Kingdom of Rohan. As such they drove the wild men back to Dunland. Over the history of Rohan there were a number of feuds between the Dunlendings and the Men of Rohan as a result, even before the Dunlendings started working for Saruman.

In that sense it might be argued that the Men of Rohan did "steal" the land from the Dunlendings, but given that it wasn't really their property in the first place they didn't have a very legitimate claim. That being said, it might arguably be the fault of Cirion for giving away territory that Gondor had long ceased to look after and had rather naturally become occupied by other people! Or perhaps the fault of Gondor's rulership in general for clinging onto the vestiges of an empire long after they had ceased being able to or interested in wielding any authority over it.