It's very similar to moving to a border town near Mexico, with the drug war going on down there (or even Phoenix)--the violence has a risk of spilling over. But that does NOT mean Americans have no sovereign right to their territory.
Obviously Mexico's official forces are not invading (though there may have been some cross-border harassment incidents, which to my mind should bring about a strong warning about frakking around with citizens of another country), but the comparison holds because whether we're talking official OR paramilitary/terrorist forces, if that territory is owned by the Federation then if you ask me, their FIRST signal of weakness in Cardassian eyes was to not send any backup with their colonists when they settled (or, conversely, restrain the colonists by force from settling in that area--not a move I would advocate unless the Cardassian claim was the first legitimate claim, but one that the Cardassians would be looking for as a clear sign of the Federation's stance). That was a sign that they did not take guarding their people seriously.
What the Federation did, by refusing to defend their people but not restraining the settlement was rather like dangling a string in front of a kitten--too tempting to refuse. Had they shown up with enough of a force that they could clearly protect their colonists, the Cardassians would've been more likely to think twice about trying to stake a claim (or if they did try, they would've gotten their noses bloodied as a result, like they did when they made a try at Setlik III and the Federation stuck up for their colonists the way they're supposed to).