Obviously this will vary from person to person - but there's something about the last third of Season Three to the end of Season Four that I just love. The end of Season Three in particular beginning with Before And After is a period of Voyager that I adore - Displaced, Worst Case Scenario, Scorpion. The series really seemed to hit it's stride in late Season Three in my opinion and this continued into Season Four. Boundaries seemed to be getting pushed with episodes like Year Of Hell, Scorpion, The Killing Game etc. The look of the show also seemed to improve significantly as CGI became more commonplace, allowing aliens like Species 8472 to be realised and also shots of Voyager herself to be showcased in a different way. I'd say this sweet-spot (again speaking personally) continued into Season Five, with Timeless, the sublime Counterpoint but with Dark Frontier I feel there was a sea change where things started to become a bit (and I appreciate how silly this sounds given the nature of science fiction) unrealistic. Getting the better of the Borg time after time, things becoming too centred around Janeway, Seven and The Doctor. For me late Season Three and Season Four is my favourite 'era'.
Season 2 was the best, followed by season 1 and season 3. All season 2 episodes were great. Even "Threshold" was watchable if you pretend that the whole episode was a nightmare Paris had after eating too much of Neelix's food. Season 1 had some weak episodes ("Elogium" and the downright boring "Emanations") but the rest of the episodes were very good. Season 3 had more bum episodes than season 2 and 1 (most notably "Blood Fever" and "Real Life") but have some excellent epiosodes like "Basics", "Future's End" and "Warlord".
Personally I Like Season 1 - mid season 4 with Season 3 being my favourite. I think things went downhill with Voyager from mid season 4 and the more frequent use of Seven of Nine. I like Seven of Nine as a character, but she overwhelmed the show and It's evident from Season 5 onwards that Seven was being used as a crutch to support a creatively bankrupt show.
Voyager started getting better when the Doctor was freed from sickbay, and Seven joined the cast. All of the shows seem to rely on the more "unusual" characters to form the foundation of the ensemble. TOS had Spock, TNG had Data and Worf, DS9 had Odo, Quark, Worf, and so on, and until a little restructuring happened VOY had The Doctor, Neelix and Kes. The Doctor was initially confined to sickbay or the holodeck, and Neelix and Kes just weren't that interesting. Neelix had a lot of potential as a character, but it just never went anywhere. His intereactions with Tuvok were supposed to be a bit of comedy, but his scenes could have been so much better. Tuvok was a great straight man, Neelix always came off obnoxious andd boring because that is all the actor could play. Additionally, Neelix's back story could have come into play, but whenever it did, it was just sub-par. "Jetrel" is among my least favorite 10 episodes... because of Ethan Phillips acting, and James Slurrrrryan, whose voice I honestly can't stand. The bottom line: Neelix had a lot of unrealized potential as an interesting alien character, but it went nowhere. Kes fared a bit better, at least the actress could act, but there wasn't much of a character there to form that "foundation". So, my opinion is that the stories started getting better when they tweaked the ensemble. The stories and scenes gradually shifted away from centering on "Harry Kim" and more on the more interesting characters... or at least the more interesting characters gave stories about "Harry Kim" more breathing room. TNG had as many boring characters as VOY, and as many so-so stories (personally I think more), they difference was in how the characters interacted in those stories. In TNG and TOS, Data and Spock stole the scenes. In VOY, that was never going to happen with Neelix and Kes.
Pretty much the same for me. 3 and 4 were the best seasons of the series. 1-2 were ok, 5-7 were good but things started to feel too "safe." Plus that was when they really started to cheapen The Borg.