IMHO the Federation space is like a block of Swiss cheese. Swiss cheese has spherical holes in it created by bubbles during processing. In the central regions of the Federation, Federation space is like the block of Swiss cheese and space not ruled by the Federation is like the bubble holes. In the outer regions of the Federation, space not ruled by the Federation is like the block of Swiss cheese and space ruled by the Federation is like the bubble holes.
Or think about physical geography on the two dimensional surface of Earth. Federation space can be like Canadian land. In the central parts of the Federation, Federation space is like continental Canada and non Federation space is like the many lakes in Canada. In the outer parts of the Federation, Federation space is like the Canadian Arctic Islands and non Federation space is like the Arctic Ocean.
Or you could compare the Federation to an island nation like Indonesia, the Philippines, the Maldives, or the Bahamas.
In modern times, the City of Rome, Italy, surrounds a foreign country, the tiny Vatican City. And it also surrounds the even tinier headquarters of the S.M.O.M., which is sometimes also considered a separate country. Italy also surrounds the tiny country of San Marino.
There are many other examples of political enclaves and exclaves.
https://en.wikipedia.org/wiki/List_of_enclaves_and_exclaves
Maybe the Federation has a rule that if a planet has a non warp capable intelligent species, the Federation won't colonize that planet or any other in the same solar system. Or maybe the rule is to not colonize any planets within 10 or 20 light years, or some other distance. This will give the intelligent beings a number of star systems they can colonize once they discover warp drive.
And if the stars within that sphere are not enough for that species, they can negotiate with the Federation for the right to cross Federation space to explore, settle, and maybe even conquer star systems beyond Federation space. I suspect that the Cardassian Union rules both a small sphere of space around Cardassia Prime containing a few stars surrounded by Federation space and a vast region of space beyond Federation borders.
And in the outer reaches of the Federation, Federation space is likely to be like islands surrounded by a sea of non federation space, both space belonging to other realms and free space that is not claimed by any realm. If a non Federation planet develops warp drive out there, they will be able to freely expand into free space but can't expand into Federation space or space claimed by other realms without having to fight for it.
And in a related topic:
In James Blish's
Cities in Flight series
They Shall have Stars/
Year 2018! (wow, that's this year),
A Life for the Stars,
Earthman Come Home, and
The Triumph of Time Earth men explore and colonize nearby stars and then encounter the Vegan Confederation, also known as the Vegan Tyranny. Centuries later Earth history books claim that the Vegan Tyranny was totally evil and that Earth men heroically defeated and overthrew it despite being outnumbered thousands or millions to one.
The local group is the term used by astronomers for a small group of a few dozen galaxies including our own galaxy. James Blish used "local group" to refer to the stars within a radius of 25, or 50, light years from Earth, with a diameter of 50, or 100, light years. Blish also mentioned the Vegan Tyranny in his adaptation of "Tomorrow is Yesterday". Mr. Scott wondered where they could go in the 1960s, because back then "space outside the local group was ruled by the Vegan Tyranny, and we all know what happened when we ran into them."
Scott didn't say that the Vegan Tyranny ruled right up to the edge of the local group, he said that the Vegans ruled space outside the local group, implying that Vegan space surrounded the local group. I believe that the Vegan Confederation is said in
Cities in Flight to have existed for 10,000 years and ruled half the Galaxy. Thus you might suppose that it spread from it's original planet many thousands of light years from Earth to rule over tens of thousands of light years, and that the local group was a tiny bubble of unoccupied space at the edge of the Confederation or actually surrounded by it.
Except that the Vegans came from Vega, and Vega was the third star whose distance from Earth was measured, back around 1840. Vega is about 25 light years from Earth, putting it at the edge of the local group if it has a radius of 25 light years, or inside the local group if it has a radius of 50 light years. So why did the Vegans explore, colonize, and conquer planets tens of thousands of light years from Vega while ignoring many of the stars closest to Vega?
Maybe the first Vegan expedition to Earth encountered Cthulhu 10,000 or more years ago and Cthulhu told them Earth was his and stay away, so they decided to never colonize any planets within 25 (or 50) light years from Earth.
Or maybe the Vegans, similar to the Federation, decided to not colonize the planets within 25 (or 50) light years from Earth so that the Earthlings could colonize them when they discovered star travel thousands of years later. Oh those evil Vegans!