This is using 20th-21st century thinking for a 23rd century situation. When Eminiar Seven attacked and destroyed the USS Valiant, they committed an act of war against the Federation. It took 50 years and the loss of many other lives to finally determine this fact. So, it is a matter of how does the Federation react to this affront? Do nothing, or simply declare it a "no-go" zone? The Klingons and Romulans are watching, so, do you show weakness or do you make an example of strength (over a technological inferior race so the downside risk is low) to show the other empires around the Federation (and probably to those within the Federation like the Andorians, etc.) that they have the means and the will to use force to protect the Federation. The starting set-up with Fox willing to start a war if intimidation fails is evidence that the Federation is putting on a show of force, up to and including, destroying an entire civilization on a planet. Harsh, but this is 23rd century politics.I can't see that genocide is justified ever.
Especially when the ships were warned (not that that justifies anyone's actions)
I mean I think its OK for the Enterprise crew not to commit suicide but actually blasting the surface of a planet is the sort of thing we think of Klingons doing in that era not the Federation