It's partly a valid complaint in that we do see the Borg quite a bit, which does kill a little of the mystery (well, for some - I never found the Borg design to be visually nightmare-inducing - even in TNG, when they were rampaging around Wolf 359. The BQ is the only thing that really looks truly scary to me.)
However, save for one tiny scout ship, there is never a point in which Voyager single-handedly takes on a Borg ship and wins. They usually run away, are helped by another powerful ship or set of ships, or get showered with magical future technology from a bitter older lady.
The fact that ordinary little Voyager sans backup was absolutely zero match for the Borg was always maintained.
Unimatrix Zero shows Voyager disabling a Tactical cube - something that's supposed to be more powerful than an ordinary cube - and warping away without chase. Q-Who? shows Enterprise disabling an ordinary cube and warping away, only to be immediately chased down by the same cube and nearly assimilated (thanks to last minute save by Q).
Hence, Voyager borg equals week and pathetic.
The funny thing here is that I think the writer's realized this and tried to explain away the weakened borg with one line in "Dark Frontier," where the queen reveals to Seven that the borg put Seven on Voyager deliberately (a sleeper cell, sort of). But that explanation only insults the character of the Borg as we know them.
Especially, when in said episode, the Delta Flyer is able to escape several pursuing Borg ships. This simply isn't believable.