Voyager was really great through the end of season 4, there were a few duds in the earlier epis but not that many. In the later seasons though after Jeri Taylor left and the show just focused on 7 all the time, the quality took a serious dive. I had also heard that B&B were very focused on getting Enterprise out there and lost their interest in Voyager so that didn't help at all. It really showed by the time Endgame rolled around. What a sorry ending to a show that had so much potential.
I actually prefer season five over season four. Season four was too much Seven. During season five the rest of the cast started to get more screentime again. But, of course, the episodes overall were better on season five from my perspective.