[Raynor]

I too doubt that the vast majority of Men knew who Gandalf really was - they believed he was an elf:

Originally Posted by The Istari, Unfinished Tales

Mostly he journeyed unwearingly on foot, leaning on a staff; and so he was called among Men of the North Gandalf, "the Elf of the Wand". For they deemed him (though in error, as has been said) to be of Elven-kind, since he would at times works wonders among them, loving especially the beauty of fire; and yet such marvels he wrought mostly for mirth and delight, and desired not that any should hold him in awe or take his counsels out of fear.
...
It is an actual Norse name (found applied to a Dwarf in Völuspa) used by me since it appears to contain gandr, a staff, especially one used in "magic," and might be supposed to mean "Elvish wight with a (magic) staff." Gandalf was not an Elf, but would be by Men associated with them, since his alliance and friendship wit Elves was well-known. Since the name is attributed to "the North" in general, Gandalf must be supposed to represent a Westron name but one made up of elements not derived from Elvish tongues.