I try to come more into the topic this time (sorry about the last one, I got a bit carried away with the associations... that's what happens when one is up too late and drinks wine to accompany himself...
).
I think some of the reasons for all this adult/modernist scorn for fantasy is based on some quite simple things.
Like associating fantasy to the fairy-tales for the children - from which every decent adult should grow up from. Now princes and princesses, dragons, valiant deeds, honour, ideals to die for, happy endings... c'mon! Nice and educative for kids but...
Like looking at the general preconception of fantasy with it's widely spread half-porn imagery from Conan Barbarian to mass-fantasy book-covers. So fantasies indeed for nerds... (wasn't it that the creator of Conan was a small and grey office clerk or something? Would fit the general scorn nicely)
Like associating the fantasy to the not-here, not-serious, not emancipatory literature. Using old forms of storytelling are initially bad for any modernist and the word escapism has been heard quite a many times as well.
And stuff like that...