As Saito S pointed out there is no analogue to raising shieds in present warships (I thought about mentioning that, but was just too damned lazy!), so your example breaks down at that point.
It really doesn't. Flooding the tubes of a ship preparatory for launch is basically a detectible defensive measure that the other sub can hear. You actually don't have to plot a firing solution or have any intent to fire, it can quite easily be just a readiness measure... Not unlike "shields" in the ST universe*. There are no perfect analogies obviously, but the analogy needn't be exact for the psychological point to come across.
Except raising shields isn't an offensive measure, which is too much to dismiss as close enough. I will grant you I wouldn't want to go into battle with out them, but there are separate detectable actions for readying offensive measures in ST. So there is no reason for the other skipper to get too perturbed, especially when he knows regulations exist that cover what the first ship is doing in response to the situation he is creating.
And I take it it's coming across, yes? It would in fact be really spectacularly unlikely for an American sub commander to assume even in a peculiar situation that one of his own ships is the enemy and to prepare accordingly, right?
In real life, maybe. I would hope (s)he might keep the possibility in mind especially if it had no downside. In Starfleet, where commandeering its ships seems a lot more common, not so much.

I mean, it's not just that the Reliant isn't communicating, which by itself is apparent cause for significant caution, you have to take all the evidence at Kirk's disposal into account. How often do such situations happen in real navies? Probably not often. And in Star Trek we can't assume what you are look at is not an illusion of some sort, which is less likely today.
I'm not talking about "residual psychological resistance." I'm talking about its being a basic pyschological component of your average navy -- and Starfleet is essentially a Space Navy -- to have an extremely strong inhibition against firing on your own ships or assuming they will fire on you.But even it there is any residual psychological resistance, that presumably is precisely why GO 12 exists.
You're certainly making the case for the importance of sticking rigidly to GO 12 (whatever it is), And here you were suggesting it wasn't really for Starfleet ships. But if it was so unthinkable, why did Saavik even bring up the regulation? Was she naive or paranoid? Clearly GO 12 exists not for the obvious potential threats so much as to ensure the unlikely ones are handled properly. As mentioned above, everyone should know the score so there would be no "offence" or misunderstanding.