The Enterprise one is a blending of the two, actually (stupid decision IMO, they should've kept the TNG effect or made a new one, not the cartoony movie streaks which are lame as anything, but I digress)
So the stretchy (and hella, cool looking!) "Rubber band effect" could be the result of the newer engines, whereas the colour trails are the result of the previous generation of engines.
The blue warp shine and the subspace entry flash are just a result of the antimatter process and since we never saw a TOS ship enter warp in that series I guess we'll just have to imagine them doing the same thing
Also, taking a leaf out of my other passion, Formula 1 and Motorsports in general, surely it is obvious that for todays racing cars the engine development and the chasis development are two separate animals.
The engine guys work on making the best engine they can and leave the stuff like aerodynamics and such to the chasis department, in the case of racing teams looking for a new deal, only coming together at the last moment to form a whole.
Applying this to starfleet I can see the whole "Transwarp project" being completely seperate to starship design, it didn't appear to be a new great power source that was going to do anything for the ship bar make it go faster.
It's more than reasonable to assume the transwarp people worked on their hypothetical new engine to the point of thinking they were ready to put it in a ship.
After a couple of test sleds showed promise, Starfleet probably elected to slot it into their newest ship off the assembly yards to further enhance its "next generation of starship design" potential.
I don't see the Excelsior being a complete, groud-up Transwarp ship, so once that proved a failure It wasn't probably too much of a stretch to slot out the Transwarp engine, unbolt the seat restraints, re-bolt Scotty's 4 screws and send it out with the latest warp engines instead.