I'm of two minds:
1) The Borg should have all been wiped out in BOBW, the Cube should have been the ONLY Cube carrying the entire Borg population.
2) The Borg should have all been killed when the Queen died, or "Scorpion" should have mentioned that the 8472 killed nearly all the Borg leaving only a small flotilla left by the time the story began.
Essentially, there never should have been a huge Borg Collective left by the time of Voyager.