Even Tolkien, who was dealing with highly fantastical and strange beings did his best to ground the environment and the characterization in to familiar tropes, i.e. he self-identified as a "hobbit" who liked "good food, a good smoke and good company." The characters have characteristics that can be connected with on a very personal level.
But the Aliens in Star Trek are human. The vast majority of the alien races in Star Trek, especially those that have been main characters are not any more different from us than Hobbits are. They might have a super power or something, but their behaviors, personalities, motivations are very human, maybe just painted in a different colour.
And personally I find many of the alien or non-human characters a lot easier to "connect with on a very personal level" than the majority of human characters in star trek.
I mean there's a lot of reasons why Spock was the most popular character in TOS.