The only prior contact with the federation was with the Hansen's as seen in Voyager. This would have introduced them to "us", but I doubt we would have been perceived as a threat, considering at the time the only people the Borg had assimilated were 3 civilians and one transport ship.
But they didn't just assimilate "3 civilians and one transport ship." -- the collective also assimilated everything the Hansens knew and whatever they could learn from their ship. While that still may not be enough to constitute a threat to them, it's a whole hell of a lot more information if they'd been taken by say the Klingons or the Kazon.