Personally, I think it was a strength of the show that the same adversaries did not show up very often and a strength of the show's premise that it kept Voyager on the move, encountering different things.
It all depends on what kind of show you want to watch. If you want a soap-operatic show that keeps returning to the same characters and material, then recurring adversaries are a good idea. But Voyager was an episodic adventure show, like the original Star Trek. In the original show, even the main bad guys, the Klingons are only featured in three episodes--they're sort of a background nuisance in a handful of other episodes, most notably "Friday's Child" and "A Private Little War."