There were plenty of other times when the command team or various members were held hostage in TOS and no-one ordered General Order 24. Particularly in "Bread and Circuses". Kirk ordered nothing was to be done. although I don't know why they couldn't beam just Spock up as he wasn't human (sorry beside the point).
Kirk was using the right order (bluff although it might have been implemented) for the right situation. He knew that the guys were terrified of destruction so he used that against them by explaining the execution General Order No. 24 in detail.
And Scott would have to go along with Kirk's order. For all he knew Spock was dead, the Emanems had a Nemesis type Doomsday device aimed at Earth. He wasn't there. He had to go along with Kirk in the heat of the battle. Although I think he would have given them a chance to surrender after the hour was up.