Nice points by Mouth and Inzil !
OK so,
some sort of contact is going on between men of Gondor and the elves. I'd think that this was much in the same sort of way that Bilbo kept in contact with the elves. Private meetings by night out in the countryside.
I have a feeling that the elves do sometimes like to wander the hills and woods, and are sometimes, like Haldir, sent to gain information. However they don't appear to make themselves publicly known in Gondor and Rohan, so avoid towns and cities, and of course can travel inconspicuously in their grey cloaks!
I think that there must have been some few Gondorians and perhaps Rohanites
, who were contacts for the elves, just like Bilbo in the Shire. Maybe they were named 'elf-friend' too? These contacts would perhaps exchange news with the elves and be trusted to keep their meetings private, if not secret, for Bilbo was well-known to 'talk with the elves'.