I've never had the impression that Kirk and scott were bluffing, and believe that Scott would have indeed carried out Kirk's order if required.
Along with Picard's "Let's make sure..." speech in Yesterday's Enterprise, this still gives me chills when I hear it:
"This is the commander of the U.S.S. Enterprise. All cities and installations on Eminiar Seven have been located, identified, and fed into our fire control system. In one hour and forty-five minutes, the entire inhabited surface of your planet will be destroyed. You have that long to surrender your hostages."
For all the great Kirk and Spock scenes that take place on the planet in this episode (and there were many), this was Scotty's finest hour. I have no doubt that Eminiar Seven would have been turned into glass, as ordered and on schedule.