Essentially, if the Borg really are "hundreds of millennia" old, like Guinan and Q claim (but how should the first one know, and why should we trust the latter?), they would have been everywhere at least a thousand times already. Their agenda just doesn't include conquering the entire Milky Way, because where would they assimilate new ideas then? They pick and choose, and after they have chosen, they may discard. There were only two times the Borg were shown possessing a planet - in "Scorpion" where the 8472 destroyed such a planet, and in ST:FC where the Borg plan involved temporally/temporarily assimilating and keeping Earth. Probably the Collective simply has little use for planets, or territory. Heck, even the very existence of "Borg territory" in VOY may have been largely a misconception by our heroes, a brief concentration of Borg activity in a particular part of the galaxy before they moved on to other places.
Timo Saloniemi