The argument often is that VOY lacked continuity not that it had no continuity, there is a subtle difference.
The premise indictaed a more serialised show, yet what we got was a largely episodic show i.e. a re-hash of TNG. Had VOY been set in the AQ exploring the borders of UFP space much like TOS and TNG some of the critisims would not have been made, as that premise would indicate a more episodic show.
But leaving aside that for now, audiance tatses where changing around that time to wanting more serialised shows.
The are some that that hate VOY, some that accept it for what it was and some who feel it could have been so much more, and some of those who critise VOY come from those who feel it could have bee more, they don't hate it per say they just feel it didin't live up to it's potential.
there are also people who LIKE the show, think it's pretty darn good the way it is, love the characters, etc.