I don't have many answers regarding why the Borg weren't assimilating the species we saw. I always assumed that the Nekrit Expanse that Voyager
crossed before encountering the Borg acted as some kind of barrier.
We know the Kazon were 'unworthy.' I don't think the Borg care about exterminating a species if they have nothing to offer for assimilation. The Vidiians would probably be avoided for the same reason. The Voth seemed to live in a city-ship, so I'm guessing they would be good at avoiding the Borg. The Hirogen I honestly have no answer about; when VGR was encountering them, Kes had supposedly thrown the ship past Borg-controlled space yet the Borg did come back.
It might have been more interesting if we'd seen the Borg try to assimilate the Vidiians for their medical technology (or maybe something random we hadn't seen - a long range transporter, a warp coil) only to find that the Phage could destroy the nanoprobes.