Then Voyager goes on to destroy entire groups of Borg ships without getting a scratch
Repeating what I said above: No - that never happened.
Outside of a small scout ship, Voyager never killed a single Borg ship on its own without outside help of some kind, much less "entire groups." "Endgame" was the only thing I'd consider overkill, but even then they were working with fancy future technology and someone who had spent years upon years dealing with and studying the Borg.
We saw a large group of Cubes just once, in "Scorpion." And the only thing Voyager did was sit there and wet its pants as they flew by.