I recently got ahold of some old FASA stuff, and it's been interesting to read some of the early descriptions of transwarp. It was never really explained in canon how it was supposed to function, or why it was superior to standard warp. FASA suggested that the transwarp engines were designed to work in conjunction with the transporter system, and that this combination would allow a ship to project a "beaming envelope" ahead of itself which would operate in tandem with the warp envelope. In effect, the ship would transport itself while simultaneously warping space around it and thus be able to boost its traditional speed faster than normal.
Mr. Scott's Guide
has a somewhat different interpretation, with the transwarp project being based in part on the interphasic rift encountered by the Enterprise in "Tholian Web." The engines were designed to replicate this rift on a much smaller and controlled scale, and the ship would travel faster in the rift than in conventional warp while the crew would experience no change in their perception of time. This system is more similar to FTL drives in Star Wars and B5.