This was really the point in which the series was at a creative and adminstrative cross-roads. Everyone knew at this point that important decisions were going to be made that would fundamentally alter the series going forward.
So, let's say you were given full creative control at this point, in between Seasons 3 and 4. Assume you have no interference from the network. What do you do?
Do you keep Kes and get rid of Harry? Do you jettison any other characters? Do you add any other characters? New villains? New romances?
Do you treat Seven's role differently going forward? Do you map out a broader strategy for getting Voyager home safely?
Does it become more serialized under your direction?
You have the keys to the car with four seasons to go. What do you do?
Early on in season 4, after an episode or three dealing with different people - low-exposure or new characters, even; just like Lower Decks or Good Shepherd (hey, let's even use those guys) - reacting in varying shades of "bad" towards their still long journey home, Voyager finds an M-Class planet fairly similar to earth. It's inhabited, scarcely, by a benevolent humanoid race and its paradise-like allure takes its toll on the crew. A good twenty-something crew members decide to stay and lay the foundation for a federation colony. Chakotay, who had an enlightening encounter with the planet's spiritual realm, elects to remain as well. Voyager departs, down 23 crew, a share of supplies and 3 shuttlecraft.
This new situation shakes up some of Voyager's rank dynamic. Tuvok is promoted to First Officer, and with his counsel and coaching, Seven takes over as chief of security instead of sidelining Harry as science guy. Both Tom and Harry get a promotion.
Neelix, in his eagerness, precedes Icheb and officially enlists. In a two-parter, he of all people is responsible for a succesful encounter with the Hirogen and Janeway declares him acting Ensign.
Kes and Seven form an uneasy relationship at first. Workforce happens much earlier, and the two of them will be the ones who remain 'outside' and work together to retrieve the crew (instead of Kim/Neelix/Chakotay) and bond over it. Kes is crucial to Seven's gradual integration and, to some extent, acts as yet another guardian figure for Naomi.
Kes is also the one who dies in Mortal Coil. Seven's probes bring her back, but it's not clean. The probes end up locking Kes in a status quo - every time she goes to sleep, her body is reset to what it was before, so while she's now functionally immortal, she's unable to form new memories as a result. A literal journey to her mind takes place, and the doctor eventually finds a way to repair her brain with the added bonus that Kes' lifespan is extended to about 40 years.
When Voyager eventually finds Equinox, things are much different from what we saw. Equinox is a much bigger ship, something along the lines of a Nebula-class successor (with Sovereign aesthetics). On board, things look ugly - after the experimentation on the nucleogenic aliens, part of the crew mutinied and the ship is adrift, most key systems disabled or destroyed due to an on-board civil war. Push comes to shove and Ransom and his friends manage to force saucer seperation from their Engineering, aiming to blow up their saucer as well as Voyager from the secondary hull, but a surprise attack from the nucleogenic beings destroys the stardrive section. Harry finds a way to communicate with the aliens in time to explain and negotiate. In an amusing twist, they consider the Equinox the offending party and the crew mere tools, so Voyager gets to save what's left of the insurgents and the aliens destroy the saucer section. Bam, we're up to 165 crew now.
This adds a few extras. The Doctor actually gets two nurses (and a semi-jealous Tom who only starts missing sickbay duty when he's no longer stuck with it), B'elanna gets a half-andorian engineer she develops playful rivalry with every few episodes - one we never see, a la Captain Boday. To compensate, every Borg child but Icheb dies in "Collective" to keep the extra count about even.
8472 never builds their silly holo-station. Fair Haven is now 19th Century London with murder mysteries instead of Janeway's ethnically offensive Holo-Vibrator.
The conflict with the Vaadwaur is a recurring theme for most of season 6 - there's a bounty on Voyager, but by the end of the season they manage to elude it and leave by virtue of the graviton catapult hurling them way beyond Vaadwaur space and another 7 years closer to earth - the original intent was to get them to the very border of the quadrant, but during the final confrontation with the Vaadwaur the catapult is caught in crossfire and starts exploding just as Voyager is launched (conveniently leaving the pursuers without a way of following as well).
Lack of Chakotay means Seven never has a half-assed relationship with him. She instead slowly and carefully approaches Harry again, and they never quite hook up even though things are slowly developing in that direction.
Friendship One never happens. Carey finishes his model in time for the last episode and gives it to Janeway to celebrate their eventual return to the Alpha Quadrant.