There is no need to be a good swimmer to be a diver - the better you drown, the sooner you reach the bottom
But seriously, I mean, there were diving costumes even in Middle ages (or so I seem to recall reading somewhere), they were made of copper and leather - copper diving bell for your head and some kind of leather hose to give you air. I mean dwarves of the mountain might have had the skill of doing such things. Only superstition kept all who would from diving there, but I think it would wear off over time - indeed, even with live dragon about, lakemen doubted his existence, so maybe, even if Tolkien haven't told us so, some generations after, lakemen started to dive for jewels after all.