I honestly think losing his brother and nephew was the worst. Sure being assimilated and helping kill many people was probably very horrific but he could in a way detach himself from that, place blame on an outside force that made him do things he normally wouldn't do.
But losing your family, the only family you have, and the only chance you'd ever had at anything reflecting a son, that's hard. Especially after they had reconciled their differences.
I don't even think losing Data ranked up there as as hard as losing his family.
I think the loss of his brother brought the reality of his age and his mortality. Even assimilation with the Borg didn't bring that up. I think up until their deaths he lived with some sort of belief that he had more than enough time to do what he wanted.
With their death he realized that perhaps he didn't, that even those who aren't taking risks on a regular basis can die, he realized he didn't have the time he thought he once had.
I think this was reflected too in his experience in the Nexus, having a wife and children. His longing for a family and the realization that he wasn't going to actually have one. That sacrafice and realization I think was perhaps the worst moment for him.