I think that photon torpedoes were off the table because of the antimatter damping field prevented their use.
That sounds like a good theory but would that not have also incapacitated the Enterprise
to make use of her matter-antimatter energy ("warp power")?
If that thing was capable to "deactivate" the onboard antimatter of the Constellation,
we could assume it did the same with the Enterprise
which then relied entirely on energizer power (or fusion power, IMHO).
Alternately, photon torpedos essentially provide a devastating and diffuse blast while the phasers have properties of a cutting beam (same story with "The Paradise Syndriome", no use of photon torpedos either). I was under the impression they tried to cut through the hull of the planet killer.
But even then, of course, firing some photon torpedos into the throat of the planet killer would have surely provided a yield of over a 100 megatons, so I do believe their antimatter had been rendered inert.
While the tactical approach doesn't seem to make a lot of sense in-universe, I believe we are looking at another fine example of an attempt to implement the "Hornblower Effect" into a Star Trek story.
Eventually, the Constellation's
dead hull is used as a fire ship
to destroy an enemy's vessel.