Why would he set the self-destruct, when he already has a better plan in place? Now, Khan is extremely intelligent, so maybe he should have been suspicious of Spock not setting the self-destruct, however, I am not surprised given his two dimensional thinking.
I mean, with the torpedoes Spock wins the engagement, with the self-destruct both parties are in a gridlock at best. Even the chance of it not working is better - Khan is more likely to take his people unsuspecting, than he is going to be willing to accept Spock's treat to use self-destruct.
Also the self-destruct that blows all the people up is one of the ideas that I can hardly imagine ever seeing in real life. It might be a good idea sometimes, but no person today, much less in the bright utopian future, would ever sign under a document mandating one. Not seeing a self-destruct plot only makes the film slightly more realistic than what we've seen before.
Not to mention than a bad guy getting his friends (not guilty of any present-day crime) back from the ship probably hardly justifies using it under Starfleet regulations.