I wouldn't call the trek fanbase "limited", TNG was pulling in 20 million viewers per episode in season 7....on the other hand, Trek haemmoraged viewers after that.
Remember, fan means fanatic.
A fanatic will stick with something they like through thick and thin.
Which means many of those 20 million weren't true fans but rather casual viewers turned on to TNG because it was the "in" fad at the time. Trek lost viewers because they were never interested in Trek before or beyond TNG to begin with. The numbers for Voyager are probably closer to how big the fanbase really is, seeing how those that stuck it out to watch Voyager would be the loyal fans