[Mithalwen]

I have wondered about this and there arecavfew possible answers. Even in our own world, not all monarchies have a strict line of succession. Rohan's culture with its origin in Eorl's military succes rather than in exalted blood lines like Gondor would probably choose the fittest to rule over the one who was most senior if push came to shove. I think Theoden instructed that they should choose a ruler as seemed best should he and Eomer both not return.

If Eowyn had survived and her brother had not, she might well have had the personal authority to rule. She certainly had the loyalty of Elfhelm who would have by virtue of his military prowess been a prime candidate to fill a power vacuum.

However to get back to the book situation, we know Theoden had no brothers or paternal uncles. He did have aunts and sisters. However it was stated that the House of Eorl ended with Eomer and Eowyn. It is unlikely that none of Theodens aunts or other sisters had children. More than likely some of the lords of Rohan were cousins in some degree to the king, though it is possible that the elder Gondor born sisters remained or returned to Gondor to marry and were thus discounted.

So I think that the likeliest situation is that membership of the House of Eorl was by male descent and that Eowyn and Eomer counted because Eomund was descended from a son of King Brego,
presumably in direct male descent

This and their being de facto adopted children of Theoden would have placed them higher than the offspring of senior royal ladies married to lesser lords. However had Eomund's line also failed, heir children may have been considered.