This may seem like out there theory to some of you, but I honestly think one of the reasons for Trek 09's success was that it had far better trailers
compared to ST: ITD.
The trailers for Trek '09 really pumped people up, I even remember plenty of non-fans talking about how they wanted to see the movie just by the trailers alone. I never heard the same sentiments from people who saw the STITD trailers. ITD trailers were nice, but they were not as epic and grand as Trek '09. That's partly why a huge mainstream audience flocked to see that movie.
Go to Youtube and compare to the trailers for Trek '09 to ITD,...Trek '09 is still stirring.
Here is a sample of Trek '09 trailers:
So I think it came down to marketing as well, they should have bragged about having Khan more instead of the big secret.