I honestly think losing his brother and nephew was the worst. Sure being assimilated and helping kill many people was probably very horrific but he could in a way detach himself from that, place blame on an outside force that made him do things he normally wouldn't do.
But losing your family, the only family you have, and the only chance you'd ever had at anything reflecting a son, that's hard. Especially after they had reconciled their differences.
Certainly losing Robert and Rene was a tremendous blow to him.
The question remains, though about whether or not he blamed himself for the destruction he caused when he was assimilated by the Borg. Being forced to live through ordering the destruction of your own fleet, without being able to do anything about it, would also be pretty traumatic - particularly to someone as devoted to his duty and to Starfleet as Picard. Even if he wasn't responsible for what happened, the Borg still struck him where he was most vulnerable.
I'm tempted to call it a toss-up between assimilation and losing his family.