I love season three (hint: the avatar) and I am in a decided minority for this opinion; or that anyway is my impression. I thought it was Voyager's strongest season overall. I feel the show's quality overall was a gradual improvement until it hit S3, followed up with the almost as good S4, and then a gradual but increasingly rapid decline. The show remained fairly interesting right up to the end, though, despite its many and ample flaws.
Don't fear, you are not alone. Season 3 was decidingly Voyager's strongest season. It did what none of the other seasons did, it had drama and fun in a very entertaining mix. Seasons 1 & 2 were high quality, but the show's dynamics were still a little too serious. Season 3 got those dynamics in a good zone. Season 4 shook things up a bit with Seven, which worked for awhile, sadly watching the end of season 4 looking to set things up for season 5 comes off badly because Season 5 was where the wheels started to come off the wagon.
When the show started the transition from Star Trek: Voyager to the Janeway, Seven & The Doctor show, was where I start losing interest.