I think that photon torpedoes were off the table because of the antimatter damping field prevented their use. As to why didn't they fire phasers down the throat, we'll probably never know. Although, we can't rule out that Decker didn't try it earlier with the Constellation. He said they fought at long-range
and if that is true, then a phaser hit on a pursuing Doomsday Machine would've been a "down the throat" shot.