It's partly a valid complaint in that we do see the Borg quite a bit, which does kill a little of the mystery (well, for some - I never found the Borg design to be visually nightmare-inducing - even in TNG, when they were rampaging around Wolf 359. The BQ is the only thing that really looks truly scary to me.)
However, save for one tiny scout ship, there is never a point in which Voyager single-handedly takes on a Borg ship and wins. They usually run away, are helped by another powerful ship or set of ships, or get showered with magical future technology from a bitter older lady.
The fact that ordinary little Voyager sans backup was absolutely zero match for the Borg was always maintained.