I know this isn't really an answer, but it depends.
Having done my share of manual labor, it takes a certain kind of strength to do things like shovel or swing an axe (or a pickaxe) all day. Having also practiced boxing, wrestling, and kickboxing, it takes a specific mix of strength and stamina to produce sustained high output, or move heavy weights. It takes yet another mix, of endurance and stamina, to trek on foot all day (see:
persistence hunting).
While any of the races can have members who can do any of the three, I would say that Tolkien painted a respective picture of, respectively, dwarves, men, and (wood) elves.
Edit: I hope that was clear. I'm finishing up some honeyed mead and on my way to bed. :-)