[Man-of-the-Wold]

At the time when Elendil washed up on the Lune and then subsequently established the Northern Kingdom of Exiles – Arnor – with its capital at Annúminas, Sauron had been in Númenor for several years. It had been centuries since Sauron had threatened the upper NW areas, when Imladris was besieged, not long after being (first?) founded by refugees from Eregion, under the leadership of Elrond (already bearer of Vilya), as well as Celeborn. During the War of the Elves and Sauron, due to decisive intervention of the Númenóreans, Sauron's forces were defeated at the crossings of Sarn Ford and what was later Tharbad. Elrond and his forces simultaneously liberated Imladris, which Sauron did not directly trouble again, although he remained well aware of it and its basic location, but also appreciated how difficult it was to defeat militarily.

He hated Elrond above all others, to be sure.

During the Third Age, generations before the fall of Fornost, the forces of Angmar had besieged Rivendell for 50 years. Again, even with significant military presence at hand, it could not be taken. Lack of detailed intelligence about the lay of the land might have worked against the forces of evil, but they were generally well aware of where Rivendell lay.

I think it might be compared to Nargothrond of the First Age, in that its location was not unknown, but the Elves guarded against anyone's getting to close to effectively do much through stealth and stratagem. Lothlórien and the Woodland Realm not doubt employed similar tactics, but Elrond likely had all sorts of ways to sense and counter not only the probes and sorties of those that would hostilly seek out his refuge, but also low-profile espionage, even as it was welcoming to many travelers with the right connections.

The people of Rivendell probably numbered in the hundreds at least. And besides the powers of Elrond (w/ & w/o Vilya) and Glorfindel, they were as capable defenders as any left in ME. They did not all, of course, live in the Last Homely House. Some had to be engaged in working the surrounding lands, to supply perishable foodstuff and so forth, which may have involved nearby hamlets of friendly men, too, who could also help with defense. And they may have had a remnant of the final Rhudaurian Dúnedain at the time of the Angmar siege.

Toward, the end of the Second Age, Gil-Galad had extended and solidified his realm and power, and indeed the Elves were quite strong under their last king by that point. It is this Elven Kingship that I feel best explains the name "Eriador," which I consider inclusive of Lindon. The Kingdom of Arnor was set up in the midst of this loosely defined and not well-populated realm. Gil-Galad should have been recognized then as a sort of overlord.