I think it made sense to exile Khan and the rest of his followers. For one thing, there would be no chance that they could escape, since Ceti Alpha V had no technological base. And if Kirk and the rest of Starfleet had been more diligent about putting warning beacons in orbit and all that, there would have been no chance of somebody else accidentally stumbling across Khan. So if that plan had worked, Khan and the Augments would have been permanently isolated, with a colony consisting only OF Augments, and no chance of innocent people getting hurt or killed.
Considering him a "criminal" based on what he did in the past would be pretty thick. He was a head of state, after all - immune to prosecution if anything!
Khan was not a legitimate government. He was a dictator who seized power by force. He didn't earn it. He wasn't recognized
as a head of state according to normal diplomatic channels. Thus he doesn't deserve any protection afforded to same.