In terms of reasoning? Nomad was crazy; Kirk didn't reason with it, Kirk just confused it and drove it deeper into its madness. Again, his MO was to treat the intelligent machine as a fallible fellow intellect, subject to hubris and pride, self-doubt and regret, and strike at the weak points of its personality.
Had Nomad been an analytical machine, it could have simply declared Kirk's facts "uncoordinated" once more, and ignored the rantings about "errors" while defining "sterilization" as only applying to lifeforms and declaring itself exempt from its own rules. But Nomad had glued random fragments of two incompatible sets of operating instructions together into a jumbled mess that was so self-contradictory that Kirk only needed to give it a random push and it fell apart. M5 was a tougher nut to crack...
Remember, Nomad was not even really a human built thinking space probe anymore (Kirk said that), it was an incredibly powerful (overpowered) space probe that cobbled itself a new programming directive based only partially on its original orders.
We're talking about a "thinking machine" that couldn't tell the difference between "James T. Kirk" and "Jackson Roykirk".