Harad is huge.
Going by the latitudes given in Letter 294, the region marked "Far Harad" on the LotR maps is actually just at the latitude of Tunisia; i.e "Far Harad" is just
North Africa!
This must be a hypothetical "furthest south" from the perspective of a northern map-maker.
In order to get beyond Harad to the actual Southern Continent, and assuming climate and vegetation are imagined as being similar to ours, you'd have to cross multiple climate zones, deserts, jungles, rainforests, plains, etc. You'd be dealing with strange tropical diseases, wild animals, hostile natives, etc all the way, and next week you'd have a whole new set of them to deal with.
Of course you could sail there. Aside from Ambarkanta map V we don't have much idea of the geography beyond Umbar/Tunisia, but assuming it's navigable (and that Third Age shipbuilding is up to the standards of the Numenoreans, which it probably isn't) you'd still need to come ashore for fresh water and food. A smaller expeditionary force could probably carry enough to last the journey, a conquering army almost certainly couldn't.
And all of this is just to get by Harad; we haven't even gotten to the Southern Continent yet.
Europeans didn't conquer Africa until comparatively modern times, and there are good reasons why. Likewise the Numenoreans - who had the greatest mariners, the strongest armies - never conquered either Harad or the Southern Continent. People from the Third or Fourth Age wouldn't be able to either (people from the Sixth or Seventh Age probably would, though
).