I'm sure Starfleet has both
sorts of orders: one for terminating a planet, another for bluffing on that. The second really needs the first in order to work...
Whether GO24 is the one about bluffing or terminating, it's more difficult to tell. We never see or hear what Scotty does or discusses with the crew; all we see is his "public face", his threats to Anan 7.
How did Fox beam down when shields were raised? Well, funnily enough, there's nothing in TOS about this being impossible! It's the beaming up
that can't be done when shields are raised. One also has to wonder how Scotty could fire phasers without dropping shields - and perhaps the explanation is one and the same: shields only stop stuff from coming in
, not from going out.
How did he bypass alarms, then? Well, he did outrank Kirk, at least in his own opinion, and at least to a degree allowing him to transport down using Kirk's hardware. He could simply have told the transporter operator to bypass the alarms for him. Or then he or more probably his aide would have had the necessary skills for bypassing them.
As for the illegality angle, this seems to be confusing "illegal" and "wrong", as well as "legal" and "right". All war is illegal: no nation's laws allow for armed intruders to come traipsing in. The Allies were breaking lots of German laws in opposing Hitler, the legal and sovereign ruler of not just his own country, but (after the fact) all those lands he had conquered. Doesn't mean the Allies were wrong.
As for international law, we don't know if such exists in TOS. Even if it does, the one Kirk's people have signed would obviously contain clauses allowing for Kirk's people to do as they please in cases like this. If it didn't, Starfleet would just wipe its collective warp cores with the paper.