Having that many head writer changeovers hurt VOYAGER, I think.
DS9 really had only one changeover, from Piller to Behr. It was different, but it didn't feel like a major change... at least, it felt like a direction that was being built toward already. It was pretty consistent, for the most part. Same with ENTERPRISE, when Braga/Berman left the reins to Coto. That was a big change, but it was heading a bit toward that direction already. TNG had its share of changeovers, about the same amount as VOYAGER. I think we tend to be more forgiving of TNG's shifts because the series itself was never really geared to have serialization. VOYAGER's premise basically made it so it SHOULD have been much more serialized, but could still be episodic. I think that is the heart of the issue.
If I were to put a single reason why this series gets the hate more than the others is because of all the STAR TREK shows, it didn't live up to its premise as much as their siblings.
STAR TREK set the standard and the mission. TNG did by doibg exactly what their says... THE NEXT GENERATION. DS9, I feel, actually surpassed their premise. ENTERPRISE, for the most part, stayed true to their premise, as well... humans basically getting their space legs, and seeing how everything started. I think it would have fulfilled it completely had we gotten 3 more seasons... it certainly was going exactly in that direction.
VOYAGER had great ideas, but didn't really follow through with a lot of them. Not saying it deserves the hate so many give it, but I can see why this is always used against them. I truly enjoy VOYAGER, just like all the rest... but frustration is a constant companion regarding the potential.