When I think of Voyager, it always seems like Kes' departure and Seven's arrival tends to split the series into two different series; and while there are more Seven episodes, I actually prefer the Kes era, since I think there was a freshness too the series (plus out of the two-parters and TV-movies I find "Future's End" (#1) and "Basics" (#2) too be the top 2 two-hour episodes) that left with Kes's departure and then we got all those old Janeway/Seven stories about their mother-daughter type relationship, plus many times I felt that the producers were just kicking back and just letting the show coast on he momentum that it had built up during the Kes era.
It really is a series of two halves, and like you I prefer the former. When it became Star Trek: Seven of Nine (featuring Captain Janeway and The Doctor) I lost interest. Seven's arc was an interesting one, but she was just overused to the point where is she wasn't the cause of the situation in the episode she was the solution (or both).
Admittedly, had Seven been a hot guy walking around in little more than bodypaint, I may have watched it a little more closely.
When looking at characters to remove, they should have stuck with the "sexy" (really?) Kim. Failing that they should have thrown out Neelix; his role as guide was at an end, so all he had was cook and morale officer, neither of which was hugely important. Kes could have taken over morale duties and become more of a nurse/counsellor character, whilst a recurring character (or nameless extra) could have taken over the mess hall.