"Absolute power corrupts absolutely."
--Captain James T. Kirk, "Where No Man Has Gone Before."
It could be said that some Starfleet admirals have too much
power, and even in the time of the Federation, the temptation to abuse that power can be too great for some (especially if they think what they're doing is for the greater good and the Federation is too idealistic and namby-pamby to do what needs to be done).
However, I think for one admiral that goes off the rails, hundreds (if not thousands) of others do not. But because quite a few admirals have
become extremists over the past century or so, it can make the admiralty look somewhat bad from a historical perspective (a few bad apples souring the entire barrel of otherwise good ones).