The Building of Barad-dûr

[Inziladun]

Hm. Well, I see a couple of possibilities.

1. There were actually two towers named Barad-dûr: one raised long before the forging of the Rings of Power by Sauron as a base of operations in Mordor. If that is considered, it can be surmised that Sauron decided to "improve" the Tower by razing the original structure and rebuilding it using the power of the One.

2. Elrond et al were merely guessing about Sauron using the One to build the Tower, and in the event mistaken.

Of the Rings of Power states that after Sauron was defeated by the Last Alliance:

The Dark Tower was levelled to the ground, yet its foundations remained, and it was not forgotten.

No mention there of why the foundations were still there, nor indeed why the Tower fell. Maybe the Alliance physically brought down the Tower, but didn't see the need to break and remove the stone foundations. After all, they believed Sauron was gone forever. How could they have known the foundations were unbreakable unless they tried to remove them? Would they have bothered to try, and seen it was impossible, and divined that the power of the One was the reason? Somehow, I doubt it.