Personally, I divide the series a little differently. Season 1 isn't too well focused on it, but there's an overarching idea of the crews blending together (it's honestly why I feel like The 37s getting held back for the season premiere ended up hurting the episode overall, as that empty cargo bay would have hit a lot harder after fifteen weeks of showing the development of Voyager's blended crew, rather than waiting another three months to give that emotional payoff). Season 2 builds on it, into the Kazon arc (for good and for ill). Seska is something of a binding factor in these two seasons, since she'd appeared a few times before leaving Voyager and then being the power behind the Kazon throne.
Season 3 is a change up, but I would still pair it together with the first three seasons, for the simple fact that at this point in the series, there does seem to be a feeling of the crew bonding and developing. You see little character arcs starting to appear, things that aren't long lasting but still continue across a few episodes with their payoff not coming in the episode they're introduced.
If season 1 was "learning to get along" and season 2 was "well, the neighbors won't get along with us, so we might as well move," then the overall theme of season 3, to me, is "learning to love the Delta Quadrant." This is the season that really builds on the idea that, whatever length of Voyager's journey, the crew is coming together and building up a connection with each other and exploring this distant part of the galaxy they're experiencing, going beyond the borders of any prior Federation ship.
The thing is, that all gets chucked out the airlock as of Season 4. Scorpion is, in and of itself, a great episode, it's in my personal top episodes of Star Trek as a whole. But it makes a turning point in Voyager's identity. Voyager is presented with a definitive choice - what matters, above all else, getting home or exploration? Because once Voyager reaches Borg space, once Janeway decides to make an appeal to the devil, once she says that she will not turn around and leave well enough alone in the name of getting this crew home... That's it. Voyager's course is set. This is no longer "learn to love the Delta Quadrant, and don't forget, you're here forever." This is "we will get home."
And then on top of that, you have the writers losing interest in the characters - in the 104 episodes that make up seasons four, five, six, and seven, how many episodes are centered on Chakotay? On Harry? On Neelix? On Tuvok? Oh, sure, occasionally, they'd get something, if they were lucky, maybe one episode per season would center on them. But that would be one episode out of twenty-six per year, and otherwise, they're there to rattle off lines of technobabble, or even just battle damage, rather than doing anything that really develops their characters as people or individuals.
While I won't argue that Seven and the Doctor are fan favorites who probably kept the show afloat until the finale, the fact is, the way that the writing handled things, they ended up sucking up all the air in the room so that no one else got the development that once was theirs. The show stopped being an ensemble, and centered around them and Janeway (who still got plenty to do, because she was The Captain™, but her character was pulled by the writing to be what they needed per episode enough that it became a popular interpretation to assume that she was bipolar - didn't even Kate Mulgrew suggest that on one occasion?).
Like, near the end of season three, after Neelix can no longer act as a guide because they've moved beyond the area of space that he'd explored, he started working on all sorts of odd jobs and training around the ship, to the point that in various forms of the potential future episodes, he was wearing a Starfleet uniform like the Maquis did. But this never happened in the main timeline, because he faded in to the background. Harry was stuck as the perpetual ensign, while Tuvok got a promotion for seemingly no reason, and Tom's promotion back to Lieutenant equally is out of left field. Everyone starts getting reduced to one line descriptions - Tuvok's the logical emotionless Vulcan, B'Elanna's the angry forehead, Harry's the naive wet-behind-the-ears-ensign, Tom's the flyboy pilot... Season four and on do not CARE about the people of Voyager beyond what they provide in any given scene. There's no real development or growth for anyone, save Seven and Doctor.
Once you get to season four in any given rewatch, you can effectively swap around episodes from various season and there is almost no difference. And while there's certainly shades of this in TNG, it is just all the more pronounced in Voyager, because TNG never proposed it would be anything different. Voyager was sold as two crews coming together, so put forward as a show about the people going on an emotional journey. But at this point, Voyager is no longer about the characters on this journey, it is just about the journey. About getting this ship and crew home. Their emotional journey no longer matters.
And that's honestly such a disappointment, because they had a good cast, good characters, and the beginnings of something with season three. But it all gets flushed down the toilet.
And, look, even after all of what I just said, I DO still love Voyager. It's the Trek I grew up on - I watched it from premiere to finale, all through my elementary school years. But I look back on it now, and I just see how it failed to live up to its potential. As much as I love what we have, I mourn what we could have as well.
I agree with most of what you have written. Unfortunately season 4 and onwards became the Seven Of Nine Show while the other characters, except for The Doctor and Janeway were shoved in the background.