The Federation does seem to have sensor buoys adjacent to the Romulan neutral zone, that the Romulan take into account.
I think it is a somewhat optimistic interpretation of the dialogue in "Face of the Enemy" that these buoys would actually allow the Federation to secure the Zone against intrusion. After all, the Romulan ship in the episode does travel into Federation space quite undetected!
Supposedly, there's a finite rather than infinite risk of getting caught if some spends long enough a time in Federation territory, or goes deep enough, but relatively little risk if one just dashes out of the RNZ area and heads for strategically less interesting parts ("Tin Man", "Visionary", "The Die is Cast").
I my own mind, the borders in Star Trek are more like walls inbetween rooms.
A wall is difficult to walk through, though. A line in space doesn't present any inherent hindrance to movement...
The difference between 2D and 3D might change quite a few truths about warfare. In 2D, the old rule of thumb is that attack is futile unless you outnumber or otherwise outpower the defender 3:1. In 3D space, it might be that defense
is futile unless you outnumber the attacker 6:1! Hence the massive war machine of the UFP still isn't enough to deter all sorts of upstarts like Talarians or Cardassians from making offensive moves - but when the Feds get fed up with that, they can end the nonsense with a very short and very decisive offense.