I suppose there were also graves of some Dúnedain in Fornost, but mainly, it was an entire city. The Downs were, well, just downs

But the main point is, I am saying even
if Fornost was haunted. I believe it actually was not, that it was only superstitiousness of the Breelanders to think it haunted. Just imagine it: a large, abandoned city in the midst of wilderness, and many graves of those who died in the Battle of Fornost a long time ago. It is only natural the village-folk would think such place haunted.