Were the Istari already old men...?

[Zigûr]

Saruman's hair and beard were black when he came to Middle-earth and slowly turned white over the millennia: "His hair and beard were white, but strands of black still showed about his lips and ears." ("The Voice of Saruman")

Frodo also muses upon how much older Gandalf looks when he returns after the nine year gap between his last visit and his final visit to Bag End.

So I think it's fair to say that while the Wizards already looked somewhat old when they arrived, they visibly aged over the course of their time in Middle-earth, albeit incredibly slowly considering how long they were around.