I think it's a combination of failures on both parts.
For Boma to be such a pain in the ass was definitely unprofessional insubordination..
But if Spock had been Kirk, he would have set him in his place real quickly, the way an human who understands human emotion would. Because Kirk understands human psychology. He'd say something like "I know you're all feeling pain at the loss of our comrade right now, and I am too, but we've got to focus on the mission. We've got to pull together and focus on being alive tomorrow." Now, i'm a bad writer, so those wouldn't be the exact words, but something like that would have helped calm Boma down a lot more than what Spock was doing, which was basically ignoring/dismissing their human emotions completely.
Spock is too unbalanced to lead at that point. It would be like Mccoy Leading, but giving in to panic and emotionalism instead of pure cold logic. Kirk has the balance, so he's the best leader of the bunch.