But clearly M-5 knew it was working in a simulator.
The point seemed rather to be that M-5 never had a clear idea of where it was working... It treated a simulation (the wargames) as reality. This suggests a childishly narrow worldview, while testing should already have pitted the computer against a wide variety of situations. Did none of those test scenarios deal with concepts like "untruth", "bluff", "accounting for human error" and "erring on the side of caution"?
Wesley might simply be saying that M-5 had handled the mechanistic routines of starship command well enough, and the wargames (plus the lead-in planetary survey mission) were the first time the computer faced complications. But that doesn't make sense from today's point of view, because odd complications should be more easily tested virtually than physically, and it's those
that M-5 would realistically have learned to handle before entering the wargames, rather than things like tactics or power distribution.
From today's vantage point, it looks as if M-5 really was a splendid success originally, meeting all the criteria in rigorous testing - and simply snapped later on. Unfortunately, the snapping happened at a rather crucial moment, but we don't need to assume that the circumstances of that moment had anything to do with the snapping. M-5 might simply have been doomed to remain sane for a limited period of time only, by design and default, what with being burdened with the memory engrams of a snapping-prone man.