Maybe that's exactly what Wesley thought Kirk was doing, just sitting back and watching... Wesley would have then thought "Why the hell doesn't he hit manual override and turn that damn computer off!?!?"
I think that's what went on - when Wesley says "what the devil is Kirk doing?", he means "why hasn't Kirk shut the M5 off?", not "why is Kirk attacking us?" As I recall, M5 shut off communications, so there was no way for Wesley to find out what was up or even if Kirk and company were still alive. At that point he assumes the ship is out of control, and moves to destroy it.
"Full phasers? What the devil is Kirk doing? Damage report, Lieutenant. Helm, course one six four, mark three."
Jim! Have you gone mad? What are you trying to prove? Break off the attack! Jim, we have fifty three dead here--twelve on the Excalibur
! If you can hear us, stop the attack!"
"Jim, why don't you answer? Jim! Answer! Come in, Jim!"
"Jim, break off your attack!"
I think the notion that Wesley thought Kirk was trying to prove something actually has some merit. It's too bad we can't ask Bob Wesley what he thought was happening and to what degree he thought Kirk was responsible.