Did the Eye ever leave Barad Dûr?

[obloquy]

But Sauron, without the ring, could not take physical form anymore.

This is from the movie, not Tolkien. Sauron did make a new body for himself when he took up residence at Dol Guldur. From The Silmarillion:

‘True, alas, is our guess. This is not one of the Úlairi, as many have long supposed. It is Sauron himself who has taken shape again and now grows apace; and he is gathering again all the Rings to his hand; and he seeks ever for news of the One, and of the Heirs of Isildur, if they live still on earth.’

During the War of the Ring -- the time covered in Lord of the Rings -- Sauron did have a physical body.