IMO - NOMAD wins in the end. Why?
- While we don't know the strength of NOMAD's full shields, the tech was more advanced then Frd Tech (Basis: The Federation didn't have weapons that could propel energy bolts at Warp 15.)
- The Doomsday Machine's weapon, while effective, was able to be defflected multiple times by the 1701's shields, thus, NOMAD would survive shots from the DM's weapon as well.
- The DM's hull being pure Neutronium, probably could't be blasted through by NOMAD's energy bolts, and the 1701's shields (per Spock could take 5 of those bolts before collapsing,
So, given the above, why do I say NOMAD wins, as opposed to a stalemate of some sort?
Because, as shown in the TOS episode "The Changeling" NOMAD has the ability to interface with, and change/program/improve alien technology (as seen in the 1701 engine room when he alters the matter/anti-matter reactor to produce a speed of Warp 14 for the 1701.) Thus NOMAD could fly in range of the Doomsday Machine, and reprogram/repair/improve and TAKE CONTROL of the automated DM.