I think the issue with this question is that it presupposes some kind of "God-believer" or "atheist" divide with no middle ground. Tolkien was a Catholic. That is a very specific system of beliefs. You might as well ask the same question of "appreciation/understanding" for other denominations of Christianity, let alone other religions and spiritualities.
Can a Hindu appreciate/understand
The Lord of the Rings? How about a Buddhist? A Muslim? A Jew? What about an Anglican? An evangelical baptist? etc.
Or, as has been stated, non-spiritual or less spiritual accounts of life: agnosticism and so forth.
The short answer is: yes. I find this quote relevant from Claudio Testi's article "Tolkien's Work: Is it Christian or Pagan? A Proposal for a 'Synthetic' Approach" in
Tolkien Studies 10 (2013):
the fundamental catholicity of Tolkiens work is not to be found in confessional elements related to his Faith, but paradoxically in the quite peculiar non-Christianity of his world, where the most authentic existential and ethical tensions involving the mere natural Man are represented.
Testi further argues that Professor Tolkien's work:
is meant neither for a single nation (England) nor a specific religion (be it Christian or Pagan), but for all of Mankind capable of sensing with their natural capabilities that beyond the Circles of the World there is more than memory
Perhaps an atheist does
not believe, or "sense" as Testi would have it, that there is "more than memory" beyond the Circles of the World, but in any event I think that the themes of the work are universal and do not depend on a particular spirituality to be understood. Similarly, I consider the theodicy (and, I suppose, theology) of the narrative to be internally self-consistent, such that while, for example, an education in Christian belief might be useful for interpreting some of the text, no specific real-world belief system has a particular bearing on the "appreciation" or "understanding" of the work.