Maybe add a few of the freed captives to the list of people who are going, at least Awyrgan and a few others that are in reasonably good health. The others can stay at the settlement under Uien's care.
I admit, littlemanpoet, you really have me stumped on who the Master is. My best guesses (Blue Wizards or the Mouth of Sauron) would appear to be wrong so far, but I don't see anybody else who possibly COULD be involved, unless it be Radagast the Brown in a very out-of-character moment.
Or is this the little-known darker side of Tom Bombadil?
