The federation is different from the klingons, romulans, etc because they don't go around conquering species, having an orwellian state complete with political police, etc etc.
That does NOT mean that 600 baku trespassing on an uninhabited federation planet can claim it as their own.
Much like cardassians, klingons or romulans trespassing on federation worlds can't claim them as their own. The federation might cede planets to them in certain conditions, rather than fight a war, though (which is about real-politik, not the laws in place).
Planets within federation space with indigenous conscious species are not federation planets.
Uninhabited planets - AKA with no indigenous conscious species - within federation space ARE federation planets. Much like uninhabited planets within romulan or klingon space are romulan/klingon planets. Trespassers don't change this fact - even when they've been around for a while, hiding.
So if say the Earth wasn't in the Federation and had established a Mars Colony or an Alpha Centauri colony then the Federation could say blow Mars up or order 'us' off it if they felt like it because the humans on it weren't native and Mars as well as Earth is in Federation territory