I agree that Spock was in the right, being doggedly determined to get them home. The other officers were being ridiculous. Decent burials? Pleading for inspiration? There's a ticking clock and death looming over them and they want to die like "men not machines?" Spock is 100% right about working to increase their chances for survival.
I don't think it all fell apart without Kirk, but it did feel more like a class misbehaving because they have a substitute teacher. Now, we've speculated how Kirk would have handled it. I don't think the behavior of the men would have been the same, because Kirk is human and would have expressed some regret over the deaths. He also wouldn't have suggested leaving two men behind and would have made it clear from the outset that he himself would stay behind so his men could live, if they couldn't get the weight all the way down. If Spock had been able to make that choice logically, imagine who he would have picked. Spock is too smart and his quick thinking might make the difference, so he'd save himself a seat. Scotty - obviously, he's vital. Now it's down to three and Bones is the ships chief medical officer. That buys him a little seniority. Mears and Bomb had less to contribute to their survival in orbit and to the crew as a whole. Now THAT would have gone over great!
Would Kirk have court martialed Boma had Kirk been in command of the shuttle? I don't think it would have come to that. Boma probably would have transferred out, either by his own design or recommended by Kirk. Unless Boma redeemed himself at the end; Kirk is pretty forgiving. In reality, s pointed out, the fact that he isn't on the bridge at the end is telling. Personally, I think he was packing.