A related thread which raises a sightly different issue than that raised here, but which speaks to the point raised by narfforc:
How many Istari came to Middle-earth?
Edit: It is quite possible that those not "in the know" about the Istari (ie the vast majority of Middle-earth inhabitants) would have made little or no distinction between the Istari and other Men considered to have magical powers, simply referring to them all collectively as Wizards.