I know in the James Blish adaptation of "The Doomsday Machine" they tried firing phasers directly into the "maw" only to have the beams bounce around like in a hall of mirrors, but I don't remember if even Blish had them do anything with photon torpdoes. Seems they would offer the kind of explosion similar to what the Constitution provided.
Or, do as they did in "Obsession"; increase the potential "bang" with a full on anti-matter bomb. Launch that down its "throat". Boom! Burp! Puke!
Yeah, earlier it "deactivated" the anti-matter in the Consti's warp pods, but the Enterprise's were still intact. Syphon off a bit, seal it in a "magnetic bottle" and fire away!