I can think of no circumstance in which the glassing of a planet inhabited by millions of civilians in a first strike could help the Federation. These billions of civilians pose no threat to the federation whatsoever - they can't turn into soldiers and build a billion ships overnight. Their genocide, however, will surely incite an extreme - and justified - response.
The glassing of a planet in retaliation may have some utility (if you aggressive klingons blow up earth, we'll blow up Qo'nos) - however, the cold war showed the limits of the "Mutual Assured Distruction" - MAD - strategy, and what lies beyond them - complete destruction, completely undesirable.
A takeover by puppet master-like parasites or a god-like alien who's using the people to build a doomsday weapon? Perhaps the people are in a suicide cult and attempting to make their star go supernova, endangering neighboring star systems. Maybe they have some omega particles and are about to render the alpha quadrant un-warpable. A horrifying disease threatening to spread off the planet and wipe out billions? Maybe two guys from alternate universes are wrestling around, blinking out our universe every time they touch, and they're hiding amongst a planetary population. Hell, maybe Gary Mitchell is back from Delta Vega and having some fun.
There are a thousand terrifying reasons for which General Order 24 might be necessary. Sure, its always best to find another way if you can, but if you can't and the chips are down, I think it'd be better to wipe out a planet then throw up your hands and allow for mass devesation on an even wider scale.