If you refer to my acceptance of such trade-offs in one case and not another, the difference is that here the Federation decided about Federation citizens whereas in the Ba'ku issue the Federation decided the fate of non-Federation citizens.
If there had been a war your government might have the right to redraw the border lines and force you to move. If oil is found under your house your government might also have the right to force you to move and compensate you for it but only your government and not any other government.
Problem is this...
The settlers of Dorvan V left Earth two centuries prior, which would have put it around the beginnings of the Federation. If they left Earth for good it could be seen as them renouncing their citizenship. If they renounced their Earth citizenship prior to the formation of the Federation, their citizenship could be in doubt.
So the Federation may have given away a world that wasn't theirs to give away...
AFAIK, Dorvan V was indeed said to be a Federation world in Federation territory. So however questionable that decision may have been (and we really have no idea, since the exact terms of the treaty were never revealed onscreen), the Federation was within its rights to cede it to the Cardassians.
And it should be noted that the Federation was willing to help the colonists move
. That is part of eminent domain even today - if the government takes your house, they have to pay you for it and assist you in finding a new home. Same story here. While I can certainly understand why the natives of Dorvan V didn't want to abandon their homes, it's not as if the Federation was leaving them to rot. It was their own idea to stay and subject themselves to Cardassian oversight. Not saying they deserved the subsequent treatment (from the Cardassians) that they got, but neither is the Federation responsible for that treatment.
As for the Ba'ku: