An interesting topic, Lal.
The only thing I can think of at the moment is that, the world Tolkien created was essentially a pseudo medieval world. The ancient (and, let's face it, the modern) world was a very violent place. Life is, to all outside viewers, I would expect, a pretty blood curdling, terrifying and nasty thing.
It's sort of essential to the realism (if fantasy can seem real to a certain extent*) that violence and blood stuff be included. Also, to make the seemingly easy task of killing goblins and Orcs more morally acceptable, I suppose they had to be pretty damned bad things to begin with.
As for the more horrific elements; Black Riders, Shelob and the rest; I think it stands to reason that there would be tremendously bad things as there are tremendously good things. Rivendel Vs Mordor. Galadriel Vs Shelob (these are all, of course, off the top of my head and I'm making no real connection between these characters, as such. Just pointing out that there are very good and very bad things).
Also, most things set in a realm of Dragons and warfare are almost always dumped into the 'gothic' genre because of the Dragons and warfare. Not everyone who reads The Lord of the Rings instantly wants to go out and slay some Orcs (although, perhaps if presented with one, the temptation may be slightly greater
)
In short... I don't know... Magic is probably the answer.
*In my own opinion, Fantasy is a better reality. Not everyone agrees. But I like to think it is.
