I cannot say when the show went wrong for the simple reason that I don't think that Voyager went really wrong in any way. Of course,weaknesses there have been during these 7 years and instead of correcting quickly, producers allowed them to settle for different reasons but in the end, it was nice to watch these familiar faces from different species live varied experiences.
As fas as I'm concerned, Voyager was a real experience - like X-Files was, long before even if I admit to have missed the last seasons - and above all, the only Sci-Fi TV program I ever watched from the premiere to the finale (even the episodes which I have zapped during the 1st airing, were seen).
Today I follow The Orville and as surprising as it may seem, Supergirl, that I lately discovered accidentally during a Saturday evening where I had nothing to do except stay in my home to nibble m&s while watching TV (funny fact, according to circumstance, I had plunged in Voyager in beginning with The Caretaker for the same reason and in doing the same thing, except that it was during a bank holiday, which fell in the middle of the week.

). But anyway, my biggest regret - and that was in watching the last crossover, Elseworlds with the Monitor's character I really took conscience of it - was the fact that never in 7 seasons of Voyager, the characters have really taken the measure of what they were doing by changing the course of Time with a big T, in creating imbalances in Universe. And moreover,
no one came to ask them for accounts or so little that Janeway and Cie got away with a pirouette and no consequence at the end.