The Federation had no claim for eminant to a planet that was already inhabiated, calimed by another race before the Federation was estalished. The Ba'ku had no representation in the Federation to fight the sizeure of their land.
If the Ba'ku had settled the planet after the Federation was formed and it was within Federation sapce then the Federation could lay claim to eminant domain.
The planet belongs to the federation as per federation law/interstellar treaties AKA under alpha/beta quadrant law, the federation does have the right to institute eminent domain.
BTW, the briar patch, before belonging to the federation, belonged to one of its member species or to someone else who gave it to the federation by treaty/etc AKA the baku settled land that already belonged to someone else.
Which means even eminent domain may not be applicable; the baku could be simply evicted. Unless you assume adverse possession took place (and there are a lot of conditions that must be met for this).