I think the reason Wesley is putting it all on Kirk is because of the control switch installed on Kirk's chair.
Could M-5 malfunction? Yes, and Kirk would then hit the switch and turn it off. Daystrom is on board to fix it. That's probably what is going through Wesley's mind.
And because neither Kirk, Spock, Scott or anyone on the Enterprise (with maybe Daystrom as an exception) ever suspected that M-5 could:
1. Re-route all control circuits to make manual overrides ineffective.
2. Put up a defensive shield making it impossible to be tampered with.
3. Draw power directly from the power plants and energy banks.
4. Had Daystrom's engrams and be able to think irrationally.
5. M-5 can ignore the captain's chair switch. Fail safe? What's that?
then there is no reason for Wesley to believe that M-5 could not be stopped. As far as Wesley's concerned, if M-5 malfunctioned then Kirk just can hit the switch and stop it.
So one way of looking at Wesley's dialogue:
WESLEY: Full phasers. What the devil is Kirk doing? Damage report, Lieutenant. Helm, course one six four, mark three.
(Why the heck isn't Kirk turning off M-5? He just let it fire full phasers at us!)
WESLEY: Enterprise. Jim. Have you gone mad? What are you trying to prove? Break off the attack! Jim, we have fifty three dead here, twelve on the Excalibur. If you can hear us, stop the attack!
(Jim! If you're trying to prove M-5 can't be trusted by letting it fire full phasers at us you're insane for letting it do it! Turn the machine off! And if you're firing phasers at us without M-5 active then you're insane. Stop the attack!)