I am one of the ones who much prefers the earlier seasons. I think season 1 started out well and the quality gradually increased until the end of season 3. Season 4 was about in line with season 3, and after that the show went downhill.
My reasons:
I thought the cast was much more of an ensemble during the first few seasons. Everyone got character development and all got some compelling stories. I liked some characters better than others, but I didn't dislike any characters during these seasons. I thought the writing was good (with the exception of them doing away with the Maquis storyline too quickly) and the acting was good on the part of all involved. The DQ races dealt with were new and interesting...and not just borrowed from TNG. The early seasons also contain two of my all-time favorite Trek episodes -
Deadlock and
Death Wish. And I also loved the Seska storyline and felt it was very well done.
In contrast, starting with season 5, the show became more and more 'The Seven and EMH Show'. I didn't think that was fair to the other characters...and I really, really, REALLY hate the EMH in any event, so that didn't help.

Starting when he got the holo-emitter, that character started going downhill, IMO...and lost alot of the best things about him, instead becoming a Data-wannabe.
The Borg were unoriginal and merely a dumbed-down rehash of TNG...and the Female Caretaker was forgotten about.
The writing for some of the characters became downright embarrassing in the later seasons - they made Chakotay into 'technobabble on the bridge guy', they did nothing with Harry Kim, they domesticated Tom Paris (and for that matter, B'Elanna Torres) FAR too much, and they didn't give Tuvok nearly enough screen time or stories. And they most certainly didn't give Janeway the screentime she deserved as captain. The captain should get more screentime than any of the other characters...but she really had too heavy competition with the EMH and Seven hogging every show.
The final blows were
Flesh & Blood (My least favorite episode in all of Trek) and, of course, the final destruction of Janeway in
Endgame....with the cherry on top being the Chakotay/Seven shipper nonsense in the last episode. By the end of
Endgame, I was so disgusted that for the first time ever in my Trek-watching career, I was actually angry about a TV show.
No...the first few seasons were much more fun for me. Until about mid-season 4 I really loved Voyager...and really until the middle of season 5 it was okay. But after that...it was no longer my cup of tea.