A Star Trek novel explained in great detail why transwarp drive was a failure.The crew suffered from micro distortions in local space causing madness.The effect could not be negated and also caused them to go partially into a different continuum.
I cannot remember the name but it was published in the time period after Star Trek III was released, possibly about within a 5 year time frame.
Can anybody remember the novel's title?
No, but I do remember the VOY episode "Threshold" that had to do with breaking the Warp 10 barrier (sometimes referred to as Transwarp) and our friend Tom Paris had some interesting side effects. I'd say what you're describing is pretty close to canon... but still grain of salt territory.