While you may prefer to resist the urge to paint any species with a single brush, in order to discuss the show you have to take its premise; and Star Trek posits that in order to reach spacefaring capabilities, a planet typically undergoes a unification of a superceding global culture. This permits Trek's hour storytelling using aliens often as allegories for our own failings extrapolated into the future; rather than depicting actual alien societies, in all their subcultural or individiul variegations, the fiction actually depicts ourselves here and now. This stems from Trek's theatrical morality play beginnings. In such a case, it is perfectly reasonable to accept the Vidiians - or any Trek race - at face value for the purposes of storytelling. Yes, they could have delved into more diversity but that would have soaked the entire hour and told an entirely different story - about aliens and characters, and not really about us. Razzle dazzle.
Of course reality is messier. This is not reality. Accepting for the suspension of disbeleif alien homogeniety of purpose, that the Vidiian global culture has by and large accepted the practice of using their starships for poaching intelligent beings for their body parts, one can hardly be faulted for "painting" Vidiians as an unmitigated threat. Just as not all contractors on the Death Star may have been at all interested in galactic politics. Some of them probably just wanted to earn a living, go home and watch the holo. But they were not innocent; because they were willing participants in the benefits of non-representational, authoritarian usurpation - and they were not acting to take responsibility for their society - they were going along for the ride. And that's what they did. Straight to Hades.
The simple fact is that we never saw a single instance of a Vidiian bucking the trend in any significant way when it came to control of a vastly powerful starship. It was not individuals the Voyager crew had to contend with. It was the starships and secret subterranean prison hospitals. While there may be a few scattered moral Vidiians, they are irrelevant forces in their own society, let alone burden anyone else with the onus of sifting them out - in a firefight.
While some agree that in general, it may have been interesting to see Trek's aliens as something more than a homogenous bunch - one can suspend disbelief long enough to pay attention to the basic play without getting hung up on verisimilitude, gritty shakycam reality a la BSG, or the tedious cynical complexity that pervades popular episodic television these days. People get enough of that from actual reality, kthx. This has nothing to do with critical reasoning capabilities nor racial attitudes. For some, TV is an entertainment - not an oracle. Nor is it a college course in politically-correct tokenism, to vainly display how condescendingly tolerant we, of all people, are. People all-so often gloss over basic principles (aware or not) to focus criticism on cherry-picked nuances. To that argument I have a two word suggestion: Stargate: Universe.