Summary execution for insubordination during battle can be a proper punishment. Especially considering she could've had a mutiny on her hands if she didn't send a strong message.
The point is, who decides which is "proper punishment"? Cain did. With nothing or nobody around who could check, countermand or correct her decisions. She made the rules, she interpreted the rules, she was the highest authority on all matters with no authority whatsoever above her.
In a normally functioning military, there are institutions who Cain has to report to, institutions who constantly evaluate her actions and could theoretically discipline her at any time. The problem is that there was nobody left who could decide whether Cain's actions were criminal or not... except maybe her XO, but she quickly killed him when he attempted to do just that. She alone decided whether her own actions were rightful. And in the end she decided that any
action she took was automatically justified, no matter what this action entailed.
So, the basic problem was that nobody
left in the universe could initiate a court-martial against Cain... not even in theory.