Since there was never any explicit discussion about the status of Constellation's
photon torpedoes, we don't really know for sure what all Decker did or didn't do with them.
Perhaps, during his initial attack on the machine, the Constellation's
torpedoes were completely ineffective, but maybe one of the phaser hits got lucky and did some kind of extremely minor damage to the machine - something like knocking off some minor piece of unrelated debris off the neutronium hull or something. So Decker may have had an unrealistic hope that phasers might have more of a chance than a torpedo, which is why he didn't launch any from Enterprise
But, I do think everything Decker did against the machine from Enterprise
has to also be looked at in light of the fact that he really wasn't mentally fit for command and shouldn't have been in the center seat anyway, and therefore all of his decisions, including not using the photons and not trying for a "down-the-throat" shot (not to mention flying into the machine in a shuttle), were all suspect.