I think you're giving too much creedence into ticket sales and money gross. Quality and reviews as well as public opinion do matter too.
I mean look at Star Trek IV, even if you adjusted for inflation it is still the highest grossing movie in the Trek franchise (other than the rebooted films); but if it was so successful why did the studio not partake after the formula of the 4th Trek movie into future installments?
Because critique-wise; the movie WAS NOT successful; it got mixed review by many and I have met a few fellow Trekfans that say next to Trek5, it's the worst movie in the series.
Why exactly do you think there was so much humor in
Star Trek V? Because
Star Trek IV made a shit ton of money. The financial success of
The Voyage Home directly influenced
The Final Frontier.
There's always been humor in Star Trek, the fact that two movies have humor in it does not shock me, and yes; like all sequels; the monetary success of Trek4 did make way for Trek5, but then comes the fact that the overall formula of the two films are completely different.
One is of a particular story and character driven narrative that involves a situation and problem that needs to be resolved quickly and with absolute swiftness, no real villain what so ever to drastically make the stakes high but the overall stakes presented is what creates drama and tension in the story.
Meanwhile the other film is a highly action and villain driven narrative where the entire story is built around making sure the villain is stopped with similar relative swiftness and with little casualties as possible. Guns blazing left and right, shifty one liners spewed quickly from the actors as they're made to carry the story along scene by scene. The tension in the story is made by the actions and situations they're placed in rather than the built up stakes expressed by the characters themselves.
The latter description of these two fits perfectly with the storytelling style of the two Trek movies before Trek4. Both Trek2 and 3 were primarily villain driven narratives that involved the villain doing things that were no good and they had to put a stop to them. Nothing wrong with that style of storytelling in Trek, if done right; you can have GREAT stories with it, Trek2 and Trek3 are good examples of this, they were great films, well received by critics and the majority of fans (Trek3 to a lesser extent).
Trek4 however diverted from this formula and went to the former description of the two films. Again, this is not a bad means for writing Trek, in fact; many of Trek's greatest stories work better as a means of this formula but alas it was not meant to be; while the movie mad a ton of money by subverting to the expectations of people who enjoyed the last two Trek films (people were still under the Star Wars/Alien heat at the time because those two movies kind of showed it was possible to make a sequel that can live up to and in some ways surpass the original) and they all went to theaters to see it.
When they did, the reviews by both Trekfans, casual movie goers and critics were mixed, some people liked it and some people didn't. What people who were disappointed with the movie and the majority complaint in every review was because of the change in formula where there was a lack of a main villain. People wanted another Action movie, but they didn't get it sadly and expectations were subverted to the point where even if they rectified it in Trek5 (they did anyways and went back to the villain driven formula) that it would not be saved anyways. It likely would have still failed regardless. Trek6 came back with a bang because they were announcing it as the last of the Trek movies with Kirk and Spock and don't look at me for one second and tell me that subtext alone wasn't enough to be the driving point to selling the movie.
In the end, actual critiques and reviews do matter and the studios DO listen from time to time; but the sad truth is, not everyone shares the same opinions as we do.
