I'd argue that TMP was a huge commerical success for Trek. When adjusted for inflation, only Star Trek XI has done better.
Yes, but is that box office sales or net profit? I said "semi-successful" earlier because TMP was an extremely expensive film to make, especially for its time.
Just going from what I've read -- when the big suits brought in Bennett, one of the only things they asked him was: "Can you make a Trek movie for less than 50 million $%&*@ dollars??"
I don't have the numbers in front of me...but TMP obviously wasn't lucrative enough to make TPTB want to stay the course.

I guess you could argue ENT jumped the shark with the whole Xindi storyline - it certainly fits the "Hail Mary" template that so many shark jumpings consist of. I rather liked most of it, but still... it was unabashedly an attempt to pull ratings out of the toilet.