I think the turning point at the S3/4 transition of Voyager was where the wheels started coming off the wagon.
Keeping Garrett Wang at the expense of Kes was a mistake. Garrett Wang was one of the worst actors I have ever seen, and I don't think he got along with the executive producers very well.
The claims by the executives that the character of Kes had 'run her course' were rubbish. A telepathic alien female with a life of nine years? The story almost writes itself.
In addition to that, Kes was a link to Ocampa, a reminder of the decision that stranded Voyager in the Delta Quadrant and who their sacrifice protected.
I believe the show may have been better suited towards maybe learning more about the Nacene (the Caretaker's species) later in the series. Maybe others members of his race remained in the Delta Quadrant. If this had eventuated, then Kes would have obviously played a significant part in such a story. This could have been a story arc across multiple seasons, possibly rivaling the weaker Borg stories.
The were other problems as well, the over utilisation of Seven of Nine and the focus on her body, the shift in the Janeway's character from a sensible Starfleet officer to 'Super-Kate', and the focus on three characters (Janeway, Seven, Doctor) rather than an ensemble as the show had somewhat been previously.
There were some great episodes in Seasons 4-7, and a few howlers, but the show was never quite the same as what is had been prior to the events of 'Scorpion'.
The show, in my opinion, hit it's zenith in Season 2 with the Kazon story arc and the smaller Vidiian story arc. Indeed, Voyager Season 2 had some of the best episodes of Star Trek that I have ever seen. I only wish such brilliance had been carried forward to later seasons.