The first three TOS films earned less than their predecessors before a total change in approach with TVH (not that I hold that movie as some kind of gold standard or anything) boosted the box office take. Mission impossible 3 was the lowest grossing in that franchise (despite being excellent) and look where they are now.
The thing is: The first three TOS movies - while still having diminishing returns - made a
shitload of money compared to their original budgets. And in fact, diminishing returns was the
norm back then: Even the original three Star Wars movies had diminishing returns!
Kelvin-Trek is in a completely different situation, in that the last movie didn't make enough money to cover it's cost during it's theatrical run. Yes, it's in the greens after all - blu-rays, merchandise etc. But that type of nailbaiting is not exactly what investors want - they could make much more money in the same time by betting on other horses.
The comparison to Mission:Impossible 3 is IMO better. But for M:I:4 they essentially completely revamped the franchise. They were even thinking about giving Tom Cruise the boot, instead they decided to have it a "team movie" (like the original series it's based on), with Cruise only
leading the team, but building Jeremy Renner up for overtaking the franchise. It's only after that was so much of a success, the next two movies focused more on Tom Cruise again. (And poor Jeremy Renner now had his next "Bourne"-type of role where he was
supposed to take the baton of a new franchise but then didn't!)
So a radical new approach for ST4 is needed. The biggest problem is probably that simply the time is running out. We're just months away from the 10th anniversary of the Kelvin franchise. People simply don't really care or remember that much anymore. If they wait another few years, it's probably easier to start from scratch with a new iteration.
It's absolutely possible, they just have to find that spark, that hook, that magic formula and a bit of luck that gets people talking about the movie and gets bums in seats. And a villain wanting revenge ain't it. I'd happily see a more serious piece of sci-fi at this point with something interesting and creative at it's heart. It can still have action and ship battles, just dial back the OTT stunts and set pieces.
I would absolutely LOVE that! But to be super honest, I think the Kelvin-timeline franchise is the wrong iteration for that. The last time you could have done something like that, was probably the TOS movies. The TNG movies already tried to be action movies, and the Kelvin-timeline movies are known for nothing else.
Their best bet for the Kelvin-timeline franchise IMO is probably a clever thriller. Yes, with a badguy and gadgets and everything. But not focused on action, but more on suspense (you can achieve that with a much lower budget!).
If you want to create a true high concept SF movie, a fresh start with new actors is probably the better option.