Dukat was always a megalomaniac, or at least a clinical narcissist. Think about it. After decades of being hated by the Bajorans and being forced to give up the planet due to intractable, violent resistance, Dukat still believed he could convince the Bajoran people to love him and be grateful for his benevolence. He always had an unrealistic sense of his own ability to triumph.
And who's to say he really understood how big and powerful the Dominion was? They may have misled him on that point.
Besides, one person's subjective opinion doesn't do anything to change the factors I mentioned before. They wouldn't have sent more than a limited percentage of their forces to conquer a faraway territory, and that limited percentage was more than a match for the UFP, Klingons, and Romulans combined. As the recent presidential election showed, all the opinions in the world can't outweigh simple arithmetic.