A deleted scene in INS which involved Riker and Troi in the library; Riker says something to the extent that the Son'a came to the Federation for assistance because the planet is in Fed space. So legally the Federation would have jurisdiction over the planet. This could explain the collaborate agreement between the Admiral Doughtery and Ru'afo. The Son'a filed all the legal paperwork and went through the proper channels seeking Federation help. The Federation would've gained technology and the methods to replicate the metaphasic particles of the planet for collaborating with the Son'a.
On the surface it seems legit. The Son'a however neglected to mention that the 600 occupants of the planet were their relatives.
Legally, the Federation cannot have jurisdiction over an inhabited planet that is not a member world of the Federation. Except in cases of war/conquered territory, or maybe some sort of special treaty situation, of course.
The very existence of the Prime Directive is an indication of how seriously the Federation is supposed to take the idea of every people's right to their own sovereignty, regardless of how technologically developed they are. To suggest that they would then be perfectly ok with a legal construction giving them 'jurisdiction' over planets that are not already inhabited, because their inhabitants are advanced enough have interstellar relations makes no sense.
Another problem here is the fact that the Federation doesn't even know up front that the Baku aren't native, which means Dougherty is conspiring to basically the most massive violation of the prime directive ever shown. In other words, blatantly illegal.
The problem with the Prime Directive Argument is, couldn't the Son'a have used it to keep the Federation from doing anything to stop the Son'a from kicking the Ba'ku off the planet? If the Son'a are really Ba'ku, why couldn't they have used that as a basis to lay claim to the planet and then keep the Federation out by saying its an internal Ba'ku matter.
It seems like the Son'a could have won by just telling the truth, which makes them pretty ineffective villains.