Saruman's Words to Gandalf and Galadriel

[Nikkolas]

"‘Saruman,’ said Galadriel, ‘we have other errands and other cares that seem to us more urgent than hunting for you. Say rather that you are overtaken by good fortune; for now you have a last chance.’

‘If it be truly the last, I am glad,’ said Saruman; ‘for I shall be spared the trouble of refusing it again. All my hopes are ruined, but I would not share yours. If you have any.’

For a moment his eyes kindled. ‘Go!’ he said. ‘I did not spend long study on these matters for naught. You have doomed yourselves, and you know it. And it will afford me some comfort as I wander to think that you pulled down your own house when you destroyed mine. And now, what ship will bear you back across so wide a sea?’ he mocked. ‘It will be a grey ship, and full of ghosts.’ He laughed, but his voice was cracked and hideous.

Saruman was decidedly unstable by this point but the underlined passage seems significant. What does he mean that the Elves and everyone doomed themselves by overthrowing him?