The brave people of Ithilien

[Inziladun]

The Nazgûl don't appear to have done much after taking the city, for a long time.
Sauron was absent from Mordor until TA 2941, occupying Dol Guldur. The Ringwraiths would seem to have just needed a place to hide while they kept a watch on both Mordor and Gondor, secretly preparing for their master's eventual return. Away from Morgul itself, those dwelling in Ithilien were probably fairly safe then.

The Tale of Years entry for 2901 states:

Most of the remaining inhabitants of Ithilien desert it owing to the attacks of Uruks of Mordor. The secret refuge of Henneth Annûn is built.

"Most" probably means women and children, with armed men remaining to resist the Orcs and keep a foothold.