Kirk never had trouble annihilating single-member species such as the Salt Vampire, Vaal, Landru or the Doomsday Machine. And the only thing that stopped him from destroying the entire species of Space Amoebae was that he could only access one individual. Nothing about this appeared to be contrary to his 23rd century Starfleet general orders.
In TNG era, Starfleet specifically wanted Picard to erase the Borg species and chided him for the failure; it also sponsored the plan to erase the Founders (issues of deniability aside) by providing the necessary resources. So that seems to cover the 24th century as well. Sure, Picard prevaricated about killing the Crystalline Entity, but he could have had sound tactical reasons for not killing his captive without interrogating it first; Kirk and Spock really dropped the ball there with the Space Amoeba.
However, the bigger the threat, the less likely that it could simply be destroyed. The Daleks seem to be exceptionally capable of bouncing back (at least in the most recent Who series), so Starfleet would no doubt have to invest massively in researching them and their weaknesses; this might well reveal means to defeat the Daleks without completely removing them from existence, and such means might well be more economical and achievable than outright slaughter.
Okay, but then at what point does the Federation consider genocide an acceptable response to deal with an enemy?
The Federation never seemed to consider genocide as a response to conflicts with the Klingons, Romulans or Cardassians.