We never learn that the prototype would not
have been workable. All we know is that it hasn't broken the old speed records of the Enterprise
quite yet when the two ships meet in ST3:TSfS.
And considering how extreme some of those records were, I don't wonder a bit... The Excelsior
under transwarp drive could have already been flown at speeds a hundred times faster than the fastest competing starship, excluding Kirk's ship
which was an unfair competitor for having gotten assistance from superior lifeforms and weird phenomena.
The idea that something about the Excelsior
or the transwarp drive failed to pan out is completely external to aired Star Trek. On the other hand, nothing is said or shown to establish that ships of the Excelsior
design would have required transwarp to be feasible. Perhaps NX-2000 was just a random ship of that class diverted from the main production run for this rare propulsion experiment, while others were being completed with conventional engines for conventional purposes.