The Dominion, as an assembly of species with a religious zeal for imposing order and control on others, struck me as the truest "anti-Federation." But that's just 1 reason I think they're the coolest villains.
Their superior technology made them, like the Borg, an enemy to be withstood rather than defeated. They also were thoroughly explored as characters through 5 seasons of Deep Space Nine.
The Romulans are a close second for me. Combining a martial (yet honorable in its way) outlook with a strategy of testing their enemies in episodes like "Balance of Terror" and "The Defector" made them very dangerous. And they're a little scary just because they're militant Vulcanoids.
I rate them second, though, because I think Star Trek's various writers left a lot more potential on the table with the Romulans than with the other major villains.
Speaking of scary, the Borg were Star Trek's 1 genuinely bloodcurdling enemy, and possibly the ones we're most likely to turn into, minus the space travel. But as a force of nature rather than an interstellar political unit, they could be used only a few times before they got mundane (see: "Dark Frontier").
The Cardassians are fascinating antagonists, I agree, with a rich history of exploiting colonial conquests, just like humans. Unlike the Romulans, they were brilliantly explored through characters such as Dukat and Garak. Their only weakness: With a technology level clearly below that of the Federation, they could never be the No. 1 enemy.
The Klingons were fun, but since most of them outside of Worf were primarily bullies, you always figured that after "Errand of Mercy" they had to get defeated, in order to satisfy not only their dramatic purpose, but also the demands of realism.
Quark saved the Ferengi, who deserve to be listed as a major antagonist as well. Throw out their sexism, and their generally nonviolent acquisitiveness makes them an interesting counterpoint to the Federation.
I'd also like to give the Vidiians a quick mention here as intelligent, and intelligently written, antagonists with a unique motivation compared with Star Trek's usual "evil empire" villain.