What makes you think Section 31 is being rational in how it determines what is or is not in the Federation's best interests?
Self-preservation? Self image? Granted, those are not necessarily "rational" but after all they "...are on the same team, they believe in the same principles that every other Federation citizen holds dear...
" (Sloan. DS9 : "Inquisition").
Do they really?
The principles which condone attempted genocide--against the Founders in DS9 and against the Tezwans in A Time to ...
, apparently achieved against numerous unknown potential threat species as mentioned in Rogue
--are not principles held dear by Federation citizens. The Federation didn't follow up the Dominion War by finishing the job that Jem'Hadar had started on the Cardassians, but rather extended an extensive decade-long aid program.
And on second thought, just because something is government sanctioned like the Tal Shiar, Obsidian Order or any real intelligence force today, that doesn’t make their actions "legal" or "moral". If 1/3 of all their cases were to become public (and evidence of them could be retrieve!) I wonder how many would stick in a civilian court of law. Like the case of Alexander Litvinenko or the Gestapo.
Having a secret police, interior ministry, or whatever that is responsible to the constitutional government of a polity is essential if that government is to exercise any control over its actions. Having an unaccountable agency present multiple fait accomplis to the government it claims to represent poses a serious threat to the government and the people its rules over.
Consider the documented attempted genocide by Section 31, the Changeling virus inflicted upon the Founders. The infection was not authorized by the Federation, but it was something that the Federation couldn't deal with even after it found out. How could it tell the Dominion that it had infected the Founders with a lethal virus and not avoid retaliation? The Section 31 gambit had a serious risk of backfiring: if the Dominion had a record of committing genocide even against merely rebellious species, what would it do to civilizations that were responsible for an attempted gencide against the Founders themselves?