For me, what set alarm bells ringing was in interviews promoting season 3 Jeri Taylor would stated that they were doing away with any form of conflict in the crew to make them more like a happy family, in turn making the show more like the original series. If I wanted happy families I would put The Waltons on. I loved the feel of season 2 (am I alone in liking the Kazon constantly threatening Voyager?). I liked the fact that Voyager moved away from their space as that made sense, but I never felt the Kazon were replaced by anything interesting. The Borg were tired and bit of a cop out really. Ditching Kes was another nail in the coffin as far as I was concerned (again controversial statement, she was my favourite character!). Like a review in TV Zone said, The Year of Hell
was frustrating as it depicted what Voyager should have been more of throughout: characters living at the edge of their abilities to survive. It seemed that despite their predicament, life on Voyager became easier and easier as the series went on, and once contact was made with the Alpha Quadrant all sense of risk and danger evaporated.
I always enjoyed watching Voyager whatever the show became, although much of season 6 is pretty painful. It definitely deteriorated over time, but in truth it's hard to pinpoint where.