Good catch there with "Birthright"!
Maybe the Klingons were the aggressors prior to the 2340s, and managed to take and hold it long enough to settle a colony...
They'd have to be. In the 2290s, the place was still "neutral" in terms of Klingon or Federation presence.
It might have been under Romulan control back then already. Klingons apparently didn't know that Romulans were in cahoots with the Feds and taking part in their secret presidential briefings and whatnot, so they'd probably agree to having Romulans mediate. (I wonder if even Chang knew?)
But it could have been under fourth party control as well, given the four prominent colors of sashes and flags there.
I have to wonder if the "conflict" was not more an unofficial conflict in which a splinter faction in the Romulan Empire struck at the Klingons, perhaps even a specific family(s) within the Klingon Empire.
Indeed, the Klingons themselves probably do something like that often enough, to their own.