Another danger of inbreeding, besides birth defects, is homogeny. After time, all of the genetic information is equality distributed, meaning that everybody is genetically just like everyone else. Inbreeding makes the population susceptible to disease, as no individual is much different that any other, and so everybody's either resistant or not
. One of the benefits of genetic diversity is that when things change, like the environment, or a disease shows up, you might have a percentage of the population that is more suited and might survive to fight another day.
But maybe this explains the elves, as they seem to resist change, and when the environment is not to their liking, they fade away into the West, unlike Men and Ents, who are more adaptable.