On the one hand TMP was smart sci-fi, but it wasn't very popular.
It was actually the most financially successful
Star Trek film prior to 2009, corrected for inflation. People had issues with its story, but it wasn't a commercial failure. The main reason it wasn't as profitable as Paramount wanted is because it was so expensive to make (especially since Paramount's creative bookkeepers lumped in the cost for all the previous, failed movie and TV revival projects in with the costs for TMP itself).
And it would've been possible to give a sequel more of a heart than TMP without needing to sacrifice the brain. To refine what they had rather than starting over from scratch.
But Star Wars was a space opera, fantasy. It was never meant to be serious sci-fi. Hell, I believe George Lucas has said as much in the past. It's a shame so much sci-fi has gone in that direction.
Preaching to the choir, man. It's annoyed me for decades that popular culture came to see
Star Wars as the embodiment of what science fiction was when Lucas himself explicitly did not consider it to be science fiction.
And what annoyed me even more was when people took that definition to heart and defensively said "It's not science fiction, it's plausible speculation" about things that really
were science fiction by the proper definition. Like season 1 of
SeaQuest DSV, which tried to be a scientifically plausible, well-researched extrapolation of the future, essentially hard science fiction, but whose executive producer insisted that it wasn't science fiction, because the public perception was that "science fiction" meant something brainless and fanciful, ignoring what the first 50% of the phrase is. And then the new producers brought in to retool it for season 2 said "Okay, now we're going to embrace doing science fiction" and what they did instead was mindless, incoherent fantasy that made
Voyage to the Bottom of the Sea's later seasons look smart. As a science fiction fan and then-aspiring writer, I was deeply insulted that that was what they thought science fiction meant -- and really, that it was what most people in the general public thought it meant.
Fortunately, the reputation of SF has improved since then. It's no longer mistaken for mindless fantasy, and it's no longer a stigmatized label that people who want to do smart shows have to distance themselves from.
Now I will say even TWOK is at least somewhat more intelligent that Star Wars from a science fiction perspective. It's weak in a lot of areas, but they made an attempt to provide some reasoning behind the story.
I don't even bother to make the comparison, because
Star Wars has never tried to be science fiction or science anything. It's sword-and-sorcery fantasy and Eastern mysticism dressed up with the semantics of space opera, along with a generous helping of WWII movies, samurai movies, Westerns, and a frisson of
American Grafitti.
I think you see some of the same parallels between Alien and Aliens. They were both great films, but I always loved Alien more. Partly because it was a bit more horror--being chased around a spaceship by a hostile alien (at least in that case it takes away the whole 'why don't they just leave' answer like you do in haunted houses). But it also was more serious. Aliens is a great action film certainly. But it's more popcorn fare. It's fun to watch, it keeps your attention and you never get bored, but it's not as high brow as its predecessor.
I don't make a comparison there either, because they weren't even the same genre.
Alien was a claustrophobic haunted-house horror movie in space, while
Aliens was a military action movie. They weren't trying to do the same thing with different degrees of quality or intelligence; rather, they weren't even trying to do the same thing at all. They took the same core ingredient but made totally different recipes out of it, like the competitors in an
Iron Chef episode. That's what makes
Aliens such a good sequel -- the fact that it didn't just try to copy the original but followed it up in a totally fresh and expansive way.