[Halfir]

In the foreword to FOTR in remarking on his dislike of allegory Tolkien wrote:" I much prefer history, true or feigned, with its varied applicability to the thought and experience of readers. I think that many confuse 'applicability' with 'allegory'; but the one resides in the freedom of the reader, and the other in the purposed domination by the author."

That statement - 'the thought and experience of the readers' - goes to the core of answering your question as to why LOTR has such a universal appeal. It is because the author did not desire to control - and here he is like Tom Bombadil - but gave us freedom via his style of writing to bring to the work our own thought and experience.

Frederick Buechner - reviewing Unfinished Tales for the New York Times wrote of Tolkien:"Perhaps it is a mark of great books as distinct from merely good ones that what strikes us most about them is less how truly they seem to echo the world we live in than how poignantly the world we live in comes to echo them."That is why LOTR is like the Flame Imperishable, whereas Harry Potter is simply a good book.