I detect two strains to this. What did Tolkien want, and how should we be looking at it.
Why did he write LotR? I don't know. Why do I have an urge to stuff the garden full of plants every year? Why do I suddenly like painting in the brightest colours I can get hold of? Why do I make up stories in my head? I would say its simply the Creative Urge. Scientifically speaking it means he had a highly active frontal lobe (also common in mental illness). Pyschologically speaking that he had secret urges to express. Classically speaking it was his Muse um...fiddling with his head. We've all done it, even cavemen did it. If any of us knew why then we'd be rich.
Yes, he spent a lot of time on this work and you could say he had an obsession with it, but this may be partly to do with his perfectionism. Maybe he had a disorder relating to OCD or somesuch, but we can't possibly say that. Maybe it was simply his form of comfort and escape. He certainly tried to intellectualise his urge over the years, many of his statements showing how he matured with age - high-minded when young about moral regneration and suchlike he grew up after a while and realised he wasn't going to change the world. Which also shows that real wisdom lies in appreciating your own insignificance in the great structure of things.
But it all boils down to a creative urge, a strong one. He didn't just work on the world he created for LotR, he attempted, and even wrote, other stories. He drew complex maps. He fiddled with invented languages all his life. he was a competent and prolific artist. he created the Father Christmas Letters for his kids (what a cool and thoughtful father, better than some plastic from Toys R Us!). He wrote lectures. He taught.
How should we be looking at it? Well since Barthes said the Author is Dead in 1968, you can look at it any ruddy way you like, apparently. In fact most of 20th century critical theory (New Criticism, Structuralism, Post-Structuralism, Reader-response) has pretty much ignored the Author and what he or she intended. Though some Marxist criticism seraches out the hidden political agenda of the writer. When the TS give lots of talks on the life of Tolkien in the hope of illuminating us, they're pretty much living in Victorian times as far as Critical theory goes. However, funnily enough, most readers want a bit of biography, want a bit of contemporary context.
If you want to look at the text in and of itself, without reference to author or source, then you need to use New Criticism. Post Structuralism will look at the readers, and the sources, but not look at the text particularly. Reader Response gets us all in a group and asks us how it makes us feel (and then we have a group hug
). But ultimately Tolkien is a special case, as are some other fantasy writers, as he's not just a novelist but a world builder, and much of what we do is to find our way around in that world he created amongst the mass of information, so maybe no kind of theory at all is more valid than another.