Those who dare enter Lorien

[Lalwendë]

If Men went into Lorien (and they were probably either too busy elsewhere or too scared to do so), then they would likely have come out of there altered and somehow 'other'. Look how the place affected the members of the Fellowship, and they were more or less made welcome there. A Man might come out at best bewildered and at worst, changed.

It's a familiar tale in folklore and mythology, the mortal who enters Faerie and comes out many, many years later (the traditional time scale seems to be seven years later, whereas he feels he has been there for just one day) quite altered. It's also a familiar motif, the idea of a place that's far too perilous to enter because of the faeries - see Stardust and Jonathan Strange and Mr Norrell for two good examples of it in modern literature.