Let's lay it down in the beginning that I'm a DS9/TNG fan and I might be biased towards them. Many here expressed their opinions about the problems with Voyager and I agree with most. Let's see my main points:
- Reset button in the beginning of all episodes. This bugs me the most.
Enterprise nailed it with the Xindi arc where with every episode, the ship was deteriorating and taking visible damage. Even DS9 played with this a little bit, always having something wrong with the station in the beginning. In later seasons of DS9 it was believable that the station was in proper condition, as they probably got the "funding" after DS9 became one of the most important pieces of "real estate" in the Alpha Quadrant. In TNG, it was also believable that the Enterprise-D was always in top condition as they were quite close to home. However, Voyager, lost in the Delta Quadrant, with limited resources, looked like it just rolled out of Spacedock in the beginning of every single episode. Except for a few cases, USS Voyager, with insufficient resources and alone, was always in a better condition than poor 1701-A ever was...
- Character diversity. Okay, in the 60s, having Spock as a regular was probably just as daring as having Uhura as a regular. By the time we got Voyager, I hoped to have more alien regulars, or at least less humans from the US/UK. Maybe it's just the European in me, but I would like to see Star Trek being something different than "Americans in Space". Alright, we got Jean-Luc the English Frenchman, who is on the borderline of being the kind of character I want to see more of. But if we need Human characters, why always US or UK? How about German, Polish, Czech, Hungarian, Italian, Dutch, Swedish, Finnish, etc. characters? Or a Russian character, who is not Russian for comic relief and does not have an exaggerated accent? Why not have South American people on board? What about an Indian character, an Arab, a Persian, a Turkish crewman? (You might call me dumb and naive, but I think it would perfectly fit Star Trek's ever-optimistic view of the future if two best friends, let's say Tom and Harry, were not European-American and Asian-American, but Israeli and Arab, respectively...) Going back to seeing it alienwise, I think DS9 did it again, by writing hell of a characters who were non-human. Kira, Dax, Odo, Garak, Dukat, Weyoun... they did an awesome job with the aliens. Then it's Voyager, and we're having almost everyone human or human-looking again.
Okay, this post is turning from "what's my problem with Voyager" into "how would my imagined Star Trek series look like", which is probably the root of the problem: Voyager was not what I expected it to be. Let's get back to the topic:
- Poor handling of supporting characters. I must compare to DS9 again, with tons of brilliant supporting characters on a huge space station. But maybe even TNG did more justice to its recurring characters (Barclay, O'Brien, Ro, etc). Then having a small ship with a crew of 150, you would imagine it's like a huge family where everyone meets everyone. Yet we mostly only had the main characters do everything, like the other 140-something people just being there so there could be casualties during an attack without killing the main crew. Okay, we had recurrings that were background people with no lines, those originally made for TNG (Barclay, Troi, Q), those who died anyway (Joe Carey), and the Borg kids who were kind of okay to have. But nobody else, really. And this disappointed me again.
- God-ship. This was probably the most annoying for me. I remember in Q Who and in The Best of Both Worlds, the Galaxy-class helplessly kneeling before one Borg cube, firing all it's got. Then in First Contact, a bunch of ships designed to fight the Borg, fighting one cube again, with little success. And then, we're having USS Voyager, a ship designed for exploration and not combat, fighting it's way through Borg space and whatnot nearly without a scratch. (Even if there is a scratch it's reset by the next episode anyway.) It gets totally maxed out with god armor and one-shot one-kill transphasic torpedoes.
Okay, so these are the things that bug me most. Other than these little things, I still love Voyager as part of one of my most favorite franchises. But it's true that it's my least favorite Star Trek series. DS9 and TNG are on the first place for me, then Enterprise - this might surprise someone, but I actually think it's a pretty good series and I really enjoyed Season 3 and 4, I was sad to see it being cancelled. TOS has another first place in my heart, but on a different scale - it's like you love your mother and your girlfriend/wife differently, you know. TOS is very much a favorite of mine, but I can't and don't compare it to its spinoffs. Voyager is, unfortunately, the least favorite.
I hope Voyager fans won't kill me now

I'm not all that good at conflict handling and I often write up then delete posts at online discussion boards because I don't want to create conflict.
P.S.: If they ever release Voyager on Blu-ray, and on every single video frame where Seven of Nine appears they photoshop her out of her catsuit and into a Starfleet uniform, I will touch myself vehemently throughout the whole last 4 seasons.
