I can't decide, really!
I think everybody gets to love the Tolkien's World through the Story first. But I can see what you mean, Thinlómien.
I know several people who love Middle-earth and are fond of Elves and Hobbits in a general way, without caring (or even remembering) too much about Tolkien's actual characters. (Those are mainly people who came to know Tolkien via the movies)
For me, there are many things I love in Tolkien's works. The world, the story and (very much) the characters who are kind of real and alive for me. (That's why I can't enjoy the changed movie-characters...).
One thing that affects me very much , and which is quite unique, is Tolkien's beautiful language - his different styles that make the different races and characters feel real and historical. (Or does that belong to "world"?)
And the more I read, the more I become entranced by Tolkien's worldview, by the timeless truths in his works and this special mixture of humour,melancholy and hope .