[The Might]

I think I can provide a pretty good comparison.
Just because Gondorians and others see Giants as mythical, inexistent creatures doesn't mean they don't exist.
We see that the Rohirrim had the exact same idea about Hobbits and Ents, but this doesn't mean they don't exist.
Only because giants are later no longer mentioned doesn't mean they no longer existed...and as I already said their place was taken by Ents in LotR, justifying their absence.
And if you see Gandalf's remark as a joke, what about the account on Beorn's origin. Clearly, by saying he descended from bears living in the Misty Mountains before the giants came, he is not trying to make any sort of "joke".