[William Cloud Hicklin]

It seems that in contemporary Gondor almost nothing was known of Rivendell; according to Boromir, when he asked about the dream-verse ("Seek for the Sword that was Broken/In Imladris it dwells") all his father had was "that Imladris was of old the name among the Elves of a far northern dale, where Elrond the Halfelven dwelt, greatest of loremasters." I don't think Gondor would have been inherently mistrustful of Elros' brother, but it does seem that there hadn't been a family reunion in recent centuries! Notice that neither Denethor nor anybody else even knew where the place was: "long have I wandered by roads forgotten, seeking the house of Elrond, of which many had heard, but few knew where it lay."

As to Lorien, one gets the impression that it was decidedly unwelcoming to outsiders; in fact, aside from Aragorn when had a Man last been admitted? Even other Elves weren't frequent visitors; Celeborn tells Legolas "Too seldom do my kindred journey hither from the North."