[Farael]

And we are forgetting one little detail... The elves that had chosen to stay in Middle-Earth were still in love with the place. Sure, they'd leave if there was no other option, but wouldn't you fight if someone wants to take over your house? even if your neighbours offer you to stay at their place, it's just not the same. So that is the "danger" for the elves... as well as the very real fear that Sauron will be able to forcibly prevent them from leaving once he recovers The Ring.

Then, for all the "free people of middle earth" minus the elves, the danger and terror is as grave as it sounds. Neither humans of Numenorian descent, nor Dwarves nor cute little Hobbits would have been allowed in Valinor. The exceptions made were made after the war of the ring, and to those characters that were most influential in the outcome. So they would have been forced to endure a life as slaves or as outlaws, not fighting a loosing battle but fighting a lost battle.