Why did the Men and Dwarves accept Sauron's Rings?

[The Saucepan Man]

I don't think the elves took off the Rings - they were hidden from Sauron and those were the only three of the rings of power he didn't touch. However Frodo later sees Galadrial wearing her ring so they still wore them

They took of their Rings when they realised that Sauron had made the One Ring, to which the other Rings of Power were subservient, so as not to be ensnared by him. The bearers of the Elven Rings were able to wear them once again when Sauron lost the One Ring.