Dougherty wanted to help billions by getting the radiation and not kill the Baku in the process (hence the holoship), then got the Son'a to agree to using transporter tags instead of assault teams that could kill Baku. Arguably he chose the lesser of two evils by sending them after the Enterprise.
Still, not getting that radiation wasn't really threatening the Federation's existence. It's not like they depended on it.
On the other hand, Leyton was convinced that the Federation needed much stronger leadership to survive the coming Dominion attack. Even if you find his methods appalling (which I do), he did have a point. Jaresh-Inyo seemed like a well-intentioned, but ultimately weak leader, especially with war on the horizon. (Alright the blood tests Leyton cherished so much turned out to be completely ineffective in the end, but they didn't really know that at that time.)
So, Dougherty did bad things for no pressing reason. That's a tad worse IMO.
And Cartwright... I guess what makes him really look extreme is that he agreed to assassinate his own president in the end. Killing the Klingon Chancellor is one thing, since he saw the Klingons as the Federation's most dangerous enemy anyway. But even Leyton refused to kill his president. Sure, we don't know what exactly was going on behind the scenes. Maybe Chang blackmailed Cartwright to continue with the plan or something after the Gorkon assassination.
Yeah, that is bad what Dougherty did. That's what happened at the start of World War II. It looked innocent enough when they were "just" removing all the Jews and relocate them so they wouldn't mingle with what they called real Germans. At first that was just what they were going to do. It got progressively worse, til finally, Hitler did the unthinkable. He called it "The Finally Solution"! Once you start down that path, it's just going to get progressively worse. This is why war and revenge are not the solution. Killing people for revenge is not going to solve anything; it's just going to pissed off more people. Like what is going on in the Middle EAst between Israle and Palestine. If this doesn't stop, one of these days one side is going to end up getting blown to high heaven, or maybe both sides will. And more time they spend fighting each other the poorer the people because of all the tax money that goes towards the war effort. Why is it so bad to sit down and talk when nobody actually dies? I guess pride, honor and races mean more than peace. When it boils down to it, it's all about race. It's like they don't deserve your mercy if they are not like you or part of your country. Who cares what happens to the Baku! They're not one of us. Maybe God wanted it this way because we're better than they are and they deserve it.