The quoted dialogue explanation (a partial one, mind you) of the invasive program has two aspects that would make it much more than a mere intellectual challenge to the Borg.
1) It was specifically "designed to overwhelm a computer's processing functions". It's not just something that makes you lock up into your study for weeks until you starve or get to your senses - it blows up your brain when you look at it.
2) It was supposed to do its fatal damage after "several hundred computational cycles". That's, what, half a nanosecond of computing? The Borg would of course have failsafes to stop them from getting stuck with an impossible task. But this isn't an impossible task - it's a frigging BOMB merely camouflaged as an impossible task, blowing up essentially immediately.
The problem with the attack wouldn't be that the Borg would have defenses against it. The problem would be that the attack would kill all the local Borg in a split second, leaving only a slim hope of propagating it further into the Collective.
Unless, of course, the attack would have a delay fuse. Perhaps the offensive part of the anomaly would only be triggered several days after insertion, not when the first segment of the Collective glanced at it. Since the attack was not attempted, we don't know how soon our heroes really expected it to work.
Timo Saloniemi
OK, but isn't it still just a fancier computer virus? If the Borg are taking this drone back into the collective, wouldn't they have anti-virus scanners for this sort of thing? Was there dialogue to the effect that the invasive program was going to appear hidden in some incredible way so the Borg couldn't detect it?
Again, look at the technological sophistication the Borg had shown up to "I, Borg." A huge civilization built on a collective computer network must have ridiculously good anti-virus defenses I would assume.