There is no such thing as a perfect situation in life.
By traversing the Gamma Quadrant, the Dominion simply got madder and madder. Look, not everybody is rational, and not everybody even holds the same concept of rationality. The Dominion detested solids, and to them annihilating any solid in their domain/sphere of influence is rational.
Besides, to use your China example, I think not going to the Yellow Sea or Sea of China in a state of cold war makes absolute sense.
I would say their cause for alarm was rationally motivated. Think Native Americans losing their lands and way of life acre-by-acre. Also factor in The Founders history and experience of being persecuted. It was their reaction to that alarm and their solution for it that was irrational.
And the Dominion trying to claim ALL the GQ would be like the Federation claiming ALL the AQ. They could claim it all they want but that wouldn't make it true. The Dominion was, by my understanding, a terrorist state - a North Korea or Iran so to speak.