Several sources mention the "treaty exploration zone". Combined with the Prime Directive (Reeves-Stevens') idea of trust worlds, I'm thinking the early UFP claimed a huge swath of space to set aside those trust worlds. The idea is that, when a world gets into space, worlds are set aside for them to colonize. In the novel, Talin IV had 48 worlds in a 25 parsec radius set aside for them. As mentioned above, the Organian treaty kinda supersedes that in at least quadrant 2, since the Klingons and UFP have to show who can develop a world better.
They were probably unduly optimistic when they founded the UFP, and nationalistic pride forbids them admitting later they couldn't actually maintain their claim, so there's a lot of nominally federation space that's not really controlled.
Keep the ideas coming.
That wouldn't contradict: the "treaty exploration zone" might simply be a collection of habitable systems in a particular region that are set aside for colonization efforts by corporations, institutions and private citizens/homesteaders. The space BETWEEN those systems may indeed remain unclaimed, much like the systems themselves could easily exist in obscure clusters between otherwise fully-developed worlds or even scattered here and there where they happened to have been discovered.
That might give you a conception for why the Romulans and Klingons keep showing up in such weird places. Federation "claimed" worlds might be scattered here and there in relatively close proximity to some worlds claimed by the Klingons and Romulans. Some of those races may think to themselves that being flanked by Federation "colonies" is a strategic liability and do something sneaky or violent to discourage colonists from settling those worlds.
And since I know it's coming, I'll bring it up early: a "borderless" Federation doesn't really present much of a problem from a defense standpoint since everyone IN the Federation lives in planetary systems anyway. You don't need to defend a vast 5000ly border around your territory; if you're worried about foreign invasion, you rely on your intelligence services to tell you if the bad guys are up to something, and if they ARE planning some trouble, you figure out what the target is and position your fleet to intercept them. An attempted Klingon invasion of Earth, for example, wouldn't be met at the Klingon-Federation border, but would probably be ambushed by a Federation task force near Tau Ceti or something. Even this wouldn't start a war, because the Klingons never actually violated Federation territory and Starfleet didn't violate theirs, and the high council would simply write it off as "A spirited brawl between warriors. What fun!"