If we believe the TNG Tech Manual, they were a swirling mass of phased particles in a big tank beneath the transporter chamber floor.
That is, dialogue specifies Barclay and the others as being in "the matter stream", "suspended in mid-transport", and that's what the Tech Manual says involves the underfloor phased matter tank, formally known as the "pattern buffer". Barclay isn't torn to pieces, he's transformed into a more malleable, "phased" form of matter that is invisible and can be sent through walls and across space.
Matter doesn't stay in that form for long; outside the chamber, it supposedly soon coalesces back into Barclay if everything goes well in a routine beam-out. But our other heroes use tricks to keep Barclay in the pattern buffer for abnormally long, in order to exorcise the lifeforms intermingled in his pattern. And the crew of the Yosemite attempted the same, only they succeeded way too well, as the plasma energies in their surroundings kept their buffer going and going and going... Until O'Brien hooked it up with the corresponding unit aboard the E-D in order to increase the range or penetration of the combined system (necessary for reaching the ship stranded deep inside the gas giant planet), and everybody ended up in that buffer tank. Until Barclay in his phased form managed to grab a phased Yosemite crew member, and O'Brien then ran the un-phasing routine on them both.
It's all there, in the breathtakingly complex technobabble of the episode. But you have to listen to it with the Tech Manual on one hand and a pencil on the other...
Timo Saloniemi