[Morthoron]

Based on what I have perused thus far, I am unconvinced that there was some sort of standing army that drove Sauron from Dol Guldur. The absence of an actual siege and breaking of the towers (which did not occur until Galadriel returned there at the end of the War of the Ring), and Saruman being relied on to supply his devices, indicates (to me, anyway) that the White Council's singular mission was to force Sauron to flee, and not to invest or besiege Amon Lanc.

The questions remain, if the White Council did indeed have an invasion force, why did they not topple the topless towers and lay bare the pits of Dol Guldur, and thus cleanse that loathsome place? Why leave such a scar on the land that years later Legolas and his kin would find it nearly unbearable to pass it by? Why not destroy it then and there so that it could not be remanned (or re-orced perhaps) with evil creatures, as it was during the War of the Ring? Rule number one in warfare regarding capturing an enemy stronghold is to either repopulate it with your own forces, or destroy it so it cannot be used again.

To me, logic dictates that the White Council did not have a strong numerical force for investment.