TMP would still have a stronger showing than most of its successors. Maybe it had more audience appeal than you think.
Overall, I really admire TMP (especially the director's cut) although I realize it's not a crowd-pleaser the way Khan is. I think the reason it may have grossed the most (besides home video not being around back then) was the decade long pent-up demand for Trek to return from the dead. Same deal with Star Wars Episode I (although TMP is a far better movie than that piece of crap). I also think had Khan not had so much competition from things like E.T. and Poltergeist it would have made more money.
As far as the NET profit, TMP can't have been the best ROI for the studio because it cost the most (outside of Nemesis I guess). It's just that the investment was clearly paid back in reusing the models and sets over time in the sequels (and TNG for that matter). I remember reading an article in Starlog magazine after TMP with a news report indicating that a future Trek movie was "doubtful" due to the disappointing returns of TMP, so it's all about ROI, not grosses.