Where did Gandalf go?

In TTT, when Gandalf is telling Aragorn and Co. about how he defeated the Balrog, he mentions something that I find puzzling. It's in Chapter 5, and it's right after he talks about throwing the Balrog down Celebdil. Anyhow, he says that:

"then darkness took me and I strayed out of thought and time, and I wandered far on roads that I will not tell. Naked I was sent back--for a brief time, until my task is done."

So where did he go? One of my friends said that he went to "hell". However, the part stating that he was sent back seems to point to the notion that he was in Valinor, but then again, why would he lose his memory? Anybody want to shed some light on this.