Descent into Hell!!! Rarrr!...Well, sort of...

[Raynor]

Given what Tolkien wrote later about "evil temple", maybe he wanted to show that these Men had been engaged in worship of Sauron/Morgoth and that it was this that led them astray and made them break their oaths

It is stated in The passing of the grey company that they did serve Sauron:

But when Sauron returned and grew in might again, Isildur summoned the Men of the Mountains to fulfil their oath, and they would not: for they had worshipped Sauron in the Dark Years.