When Spock and Khan were talking to each other on their bridges, and Khan said he would knock out life support and take back his crew, why didn't Spock just set the self-destruct for a short countdown (10 or 5 seconds), on pause, and say that if Khan didn't surrender the Vengeance immediately, he would destroy the Enterprise himself, along with Khan's crew? Khan had superior firepower, but the Enterprise still had shields, and it's unlikely Khan could damage them enough to completely disable them before Spock activated the countdown - Khan may have surrendered, given that threat to his own crew?
I don't think Khan would surrender. It wasn't in his nature.
Then there's the possibility that he calls Spock's bluff, destroys the Enterprise for the Hell of it and then is wondering the galaxy in a floating WMD platform that is three times faster than any other ship in the Fleet.
I think Spock made the right call.