The Genesis torpedo was clearly not ready to detonate since it was on a "build up", so it couldn't have been prematurely triggered in the process. Otherwise Khan would have rushed it himself.
It's quite possible that the Genesis device becomes lethal the moment the final button is pushed, and has to be left alone for an additional few minutes to also
become useful (i.e. be capable of creating new worlds) in addition to already being lethal. If the sensors of the Enterprise
already read the "Genesis waveform", it makes sense that this deadly thing is being kept bottled up and its release would be fatal.
Indeed, we don't know whether the build-up sequence ever finishes, or whether the Reliant
just happens to blow up, triggering a premature release of the Genesis wave (and thus contributing to the instability of the end result).
As for Khan, he probably understood next to nothing about the device. He might have preferred to blow up the Reliant
on Kirk's face, but he didn't know how to scuttle the ship. So he used the weapon he had already primed and learned to detonate.
Note how in "Space Seed", when intruder control gas is released and all is lost, Khan reaches Engineering - but apparently wastes several precious minutes looking up a way to slowly
scuttle the ship, allowing Kirk to catch up. It's pretty natural for a superman to fail to study in advance the measures needed at defeat...
Khan doing there at the conclusion of "Space Seed", though? Was he really committing suicide of a particularly spiteful sort? Or did he have a plan of blackmailing Kirk with the slow self-destruct? He certainly failed to make use of such a plan!)