My understanding was that until that moment the Conspiracy was not quite sure, but they were able to pin down the source of the Baggins mysteries. They list a number of facts they based their hypothesis on:
- Bilbo disappeared into thin air at his party
- Bilbo disappeared in a similar fashion to avoid the SBs, and Merry noticed a small shiny object involved
- Bilbo brought home treasure and who knows what else
- Gandalf probably has something to do with it
So they could piece together that Bilbo must have brought something back that enabled him to pull off the disappearing trick. Given that Gandalf traveled with Bilbo and was a close friend - and generally knew a lot more about magic stuff - they probably were in a state of heightened alertness during his visits. Is it so unnatural for Sam to be listening in when Gandalf came to spill the beans? I bet you they'd be trying to bug Gandalf since they started the Conspiracy.