I think that Shelby, after a lifetime of studying the Borg, thought she could out-Borg them. That she would take the best of their Collective strength MO and give it to her own people, and use it to combat the evil that was ultimately studying her right back. Coupled with key advisors encouraging her on…and maybe adversaries having their own successes with salvaged Borg tech and networked systems (we need to develop AI before the Chinese develop their own)…and with Starfleet being better at routing out AI threats and hacks and whatnot after PRO….she didn’t see it until it was too late. I mean, she was a 4-5 star admiral at this point; she was probably used to successes and thinking things were different in this new normal.
Also, it was an innovative new attack. The ships or crew weren’t invaded by nano probes, the crew themselves were hacked by genetic manipulation.