Khan was guilty of attempted murder and attempted hijacking of a Federation starship. But in the Federation's eyes, those wouldn't be grounds for execution. About the only crime that was threatened with death was visiting Talos IV.
Should Kirk have confined Khan and his followers to the brig and ferried them to a starbase for trial? Perhaps... But OTOH, it may have not been expedient to do so. A class M planet not claimed by anyone? Seemed a little peculiar to me, but given what little resources were given to Khan and his followers, one could consider it a kind of penal colony. Cut off from the rest of humanity, with no means of escape. And I'd have expected a probe would be left in the vicinity to monitor things like keeping away visitors.