At the time the decision to destroy Nero is taken, he isn't a threat.
And how were they so sure he wasn't a threat?
They weren't.
Well, I asssumed it was because their sensors told them he wasn't a threat. That much was suggested in dialogue. If they thought he was still a threat, they could have shown that in dialogue instead. Overall, any response is a judgment call. They didn't know that there were no children or innocent prisoners on board the Narada either (although that information might have been gleaned by Spock from the computers). On balance, I think that the way in which the summary execution was presented was a departure from the Federation principle we saw TOS.
Lord knows they have reasons to hate Nero but that kind of tit for tat killing leads to further killing - we see it all over the world today. It's missed opportunity for the writers to show that the Federation doesn't hold with that sort of thing. The Federation is liberal. I'm uneasy at any suggestion that they want to make it less so quite so casually or that being liberal is intrinsically bad or weak.
The sense of satisfaction that both Kirk and Spock display when they decide to destroy the ship is also rather disturbing. I'd have been happier if they had established in dialogue that the ship was still a threat and salivated less when they did the deed.