Much like the series, every season had its ups and downs. I wasn't particularly crazy about season two but in hindsight seasons 1/2 came the closest to being the kind of series I would have wanted out of VOY. There was an attempt to have a humanistic, character-driven approach although not always succesful. And for some reason whenever it is revisited in episodes like Shattered or Fury it gets to me via the nostalgia.
Season three was the worst. Jeri Taylor seems like a nice enough woman but she was in charge of two of the franchse's weakest seasons--this and TNG-7. The stories were weak, had way too much technobabble, were too silly or campy, the DQ aliens were pitiful, Q was misused. Episodes like Sacred Ground, Rise, The Darkling, Favorite Son, Coda, False Profits, The Q and the Grey etc were disappointing and almost drove me away from the series.
That said, I agree season four is the show's best although I think I expected a bit more of a renaissance than we got after Scorpion. I always liked Brannon Braga and his style of storytelling. Season four wasn't perfect but it was more consistent than other seasons. Seven was an interesting addition, the telefilms were entertaining, they had a nice mix of aliens(the Borg, 8472, the HIrogen, the Krenim). And a lot of the stories were entertaining if not high-minded. That isn't to say there weren't some flops like Vis a Vis, Demon, Unforgettable. Hope and Fear didn't quite go far enough. It played it too safe.
I thought season five was a step down. I didn't care for how they used 8472 in "In the Flesh". I thought Once Upon a Time, The Fight, The Disease, Course: Oblivion, Nothing Human, Gravity, Juggernaut, Extreme Risk were either weak or poor and not surprisingly they were usually the rare vehicle to showcase the lesser characters-Chakotay, Kim, Torres, Tuvok, Neelix. I just wasn't overly impressed by this year's offerings with few exceptions.
I might be one of the few who actually enjoyed season six. It isn't going to go down as one of Trek's best and it vascillates quite a bit in terms of quality. There is a definite pattern of a good episode followed by a weak one--Pathfinder(good), Fair Haven)poor), Blink of an Eye(good), Virtuoso(poor), Memorial(good), Tsunkatse(average), Spirit Folk(poor), Ashes to Ashes(decent) etc. I have fond memories of this season and I thought it ended strongly with Unimatrix Zero.
I didn't particularly care for the Paris/Torres romance or the constant harping on the Doctor as an equal which went overboard in Virtuoso. By this point in the series I was perfectly satisfied with the writers focusing on Janeway, Seven and the Doctor since the other characters weren't interesting and usually featured in the weakest episodes of the series.
Season seven was alright not great. I've never been a big Biller fan and thought he was no worse than others as a showrunner. I thought there were too many pointless or silly outings-Q2, Nightingale, Friendship One, Natural Law, Renaissance Man, Drive--when they seemed to neglect ending the show on a strong note with a satisfying homecoming resolution. I thought Unimatrix Zero was horribly wasted. I didn't care for C/7.