Season 2. Season 2 is where it went wrong.
Season 1 actually had a handful of episodes that dealt with the uniqueness of Voyager's situation. Sure, it was a mistake to make things as neat and clean as they already were at the end of the pilot, and having the second episode ever be a mostly routine spacial anomaly plot with a more relevant b story didn't help. But there were also signs of life in episodes like "Prime Factors" and "State of Flux", even if the latter is hurt by the useless and boring villains Seska and the Kazon ultimately became in season 2. The Kazon sucked even in season 1, but there were a couple episodes with the Vidiians, and they had potential to be interesting. Hell even Neelix got something of a standout episode in season 1 with "Jetrel", and Neelix is second only to Harry Kim on my list of most hated Voyager characters.
Looking at a list of season 2 episodes is like looking at a list of my most hated episodes ever. "Elogium". "Non Sequitur". "Twisted". "Parturition". "Resolutions". F'ing "THRESHOLD". No season of Trek is immune from having its share of failures, but this is just pushing it. "Death Wish" and "Lifesigns" are about the only two from this year I'd ever sit through again without a gun to my head, and "Death Wish" could have happened on any modern day Trek show. I'll throw in "Cold Fire" too because it actually gave Kes something to do for one of half a dozen or so times before the show cruelly threw her character under a bus.
After that the show seemed to level off. It's harder for me to judge it objectively once it became the Seven of Nine Show, because I can't for the life of me stand Seven of Nine, but I'll definitely grant that any season after 4 has more episodes I'll willingly sit through than season 2 managed.