Did Gandalf make a mistake at Isengard ?

[Inziladun]

When Gandalf first arrived at Isengard to survey the scene and get news from Treebeard, he did not, in fact, know that Saruman was in possession of a palantír yet. For that reason, I can't fault Gandalf for not ordering Treebeard to restrain Wormtongue upon his arrival and not allow him into Orthanc.
I can see the potential for harm there, but even if Saruman had tried to do what you suggest, I don't think Merry and Pippin would have been in any real danger at that point.

'Oh yes, I forgot about (Wormtongue),'said Pippin. He did not get here till this morning.'

TT Flotsam and Jetsam

Even if Wormtongue had told Saruman of the hobbits' presence, as he no doubt did, and Saruman immediately contacted Sauron, I don't know if even a Nazgûl would have had time to get to Isengard before the arrival of Gandalf et al.
Also, would the Ents have allowed a single Nazgûl to make off with Merry and Pippin?