Rather than hook the M-5 unit up to the Enterprise and give it "varchully unlimited power!",
Starfleet should have tested M-5 by linking it to another computer-- a virtual simulator that the M-5 thinks is the real deal. The Enterprise, outer space, and other starships would all be simulated in the virtual environment.
Starfleet officers would log on and play the game as if they were on their real ships, and only M-5 wouldn't know it was a virtual environment. Then when the unit went crazy and started killing people, all the lessons could be learned and nobody really dies.