A deleted scene in INS which involved Riker and Troi in the library; Riker says something to the extent that the Son'a came to the Federation for assistance because the planet is in Fed space. So legally the Federation would have jurisdiction over the planet. This could explain the collaborate agreement between the Admiral Doughtery and Ru'afo. The Son'a filed all the legal paperwork and went through the proper channels seeking Federation help. The Federation would've gained technology and the methods to replicate the metaphasic particles of the planet for collaborating with the Son'a.
On the surface it seems legit. The Son'a however neglected to mention that the 600 occupants of the planet were their relatives.
Legally, the Federation cannot have jurisdiction over an inhabited planet that is not a member world of the Federation. Except in cases of war/conquered territory, or maybe some sort of special treaty situation, of course.
The very existence of the Prime Directive is an indication of how seriously the Federation is supposed to take the idea of every people's right to their own sovereignty, regardless of how technologically developed they are. To suggest that they would then be perfectly ok with a legal construction giving them 'jurisdiction' over planets that are not already inhabited, because their inhabitants are advanced enough have interstellar relations makes no sense.
Another problem here is the fact that the Federation doesn't even know up front that the Baku aren't native, which means Dougherty is conspiring to basically the most massive violation of the prime directive ever shown. In other words, blatantly illegal.