This really makes the Borg less frightening as enemies. Genocidal maniacs on a rampage are dull; infinitely patient half-machine monsters intent on helping mankind reach a higher plane of existence are a much more sinister threat.
Also, it sounds rather dimwitted that humans or Feds would be an exceptional threat to the Borg. They've no doubt swallowed bigger catch whole in the preceding millennia; in VOY, we saw them in major assimilation action, establishing their anti-human antics as mere love pats.
Why would something as old as the Borg change merely because it met a bald human and then a redheaded one?