I think that the decision "sequel or not" is primarily made based on the "plan" for the sequel, especially in an already established franchise. There are films that got a sequel even though they had horrible box office, because studios liked the concept and approach of the sequel plan. There are films that never got a sequel even though they had fantastic box office, because nobody was able to come up with a plan for a sequel that satisfied the studios.
Superman Returns did fine, but the concept for the sequel wasn't good.
Pretty much this. Superman Returns had a production budget of 260 million and made just under 400 million worldwide. While it's a good movie it's a poor superhero movie. Compared to Batman Begins which made less on a smaller budget. It inspired confidence to see a sequel. While Superman did not.
This is a quote from 2008 I found on IMDB by Warner Bros. Pictures Group President Jeff Robinov, "Superman Returns didn't quite work as a film in the way that we wanted it to. It didn't position the character the way he needed to be positioned. Had Superman worked in 2006, we would have had a movie for Christmas of this year or 2009. Now the plan is just to reintroduce Superman without regard to a Batman and Superman movie at all." This is after The Dark Knight blew up at the boxoffice.
Star Trek has nothing to worry about it will get a sequel. Most people just had higher expectations for it is all. Look at Fast and Furious 6. It's already made $500 million worldwide. Trek should be capable of that kind of business.