Sisko basically nukes a planet in 'For the Uniform', and that was just a police action against a bunch of poorly-equipped colonists. And Starfleet Command okays it after the fact.
However, that was more a case of the ends justifying the means. It was retaliatory, none of the colonists are killed, and there's a nice facile conclusion whereby both sets of colonists simply swap planets.
There also seem to be all sorts of bans on the really nasty stuff, like Thaleron radiation and the isolytic weapons used by the Son'a. Why didn't the Dominion basically nuke Federation planets like Betazed? It might be out of character for the Feds, and perhaps the Klingons, but not the Dominion. Therefore, there must be technical or ethical reasons for them not doing so.