It could very well have been that at the time the writers crafted the Doomsday Machine story, photon torpedoes weren't widely used in the series. Thus they didn't think of it and at the time, it just didn't dawn upon Roddenberry and others. I can't recall, but was there ever an episode where the Enterprise employed a thermonuclear warhead? In Balance of Terror, the Romulans dumped one overboard. It would have been an interesting idea for the ship to send one down the maw of the Doomsday Machine. The vessel would probably detect just a crumb of matter and go about ingesting it without realizing the danger. The same principle with the Constellation being ingested without the engine overload being detected.