I don't know if 'wasted potential' is the best way to describe what happened with Voyager, although it definitely didn't reach its fullest potential, at least based on the original series bible (of which I have personally seen a copy). As per its original concept/premise, Voyager was supposed to be more along the lines of DS9, but what it ended up being/becoming was more akin to TNG; another thing that hindered the series from living up to the fullest potential of its concept and premise, as per the series bible, was the decision to more or less shove the conflict between the Starfleeters and the Maquis - which was an essential part of that concept - into the background to the point that it became non-existent after the first couple of episodes.
The series also didn't delve into the characters as much as its original concept and premise, as per the series bible, indicated that it would, which is a byproduct of it beocming more like TNG and less like DS9.
Regarding the Borg, they do work best in smaller doses, although given that they originated in the Delta Quadrant, it was somewhat inevitable that the writers would bring them into the series at some point. The one thing that I think I personally would've done differently with regards to the Borg was to 'seed' evidence of their presence throughout the quadrant and the series, with them only encountering the ship and her crew in a handful of 'major skirmishes'. Bringing in Seven was a good idea in concept, but it probably would've worked better in practice if the audience hadn't be 'de-sensitized' to the Borg by the time that it happened. Speaking of Seven, one thing that I do think I would've changed regarding her arc was to have introduced her (either named or unnamed) as the Borg's 'representative' during the events of First Contact in place of bringing in the concept of the Borg Queen, which, although neat in theory, doesn't quite jive in practice with everything we'd previously been told about the Borg as a 'species' up to that point. It would've given her character more emotional 'oomph' and let us, as an audience, connect more with her if we first saw her as the 'leader' of the Borg and then saw her liberated, especially if/once we started to get hints as to her personal history.