Which is what we've seen before, so if I can suggest a slight change.
The rival isn't a enemy, more a competitor. A competitor for new members, new worlds to colonize, new trade partners, new natural resources to drive and expand their civilization.
Not particularly looking for war with the Federation, but will use limited force if the Federation deliberately get in their path.
We first hear of them on the new series when a Federation member cancels their membership, in order to freely join the rival.
.
Said competitor could even be (if we wanna throw a sort of twist into it) a throwback kind of organization... one that looks on monetary and business gains, rather than exploration and such. Now, I'm not necessarily talking about the Ferengi. I'm talking about an organization from Earth that somehow has managed to exist since the founding of the Federation, and has eventually managed to get enough resources to have a force that could rival StarFleet. But, knowing that the Earth they remembers from history back no longer exists, they have founded their own world base on another planet. They set out with their own ships.
Whereas the Federation works toward the betterment of all, this corporate entity works to make its profits, forge monetary business trade with other worlds, forms protectorates (and alliances where possible). They don't feel bound by the Prime Directive, but they're not necessarily out to corrupt or enslave worlds.
The Federation may take a dim view to this corporate competition for the trust and membership of worlds, but it's not like the Federation can really do anything about it. This organization operates outside of Federation law. The forces they have can actually contest whether or not it is "Federation" space, and the Federation may have a hard time trying to press home any legality of the Prime Directive to this competitor.
It kinda becomes a "he who has the most planets, rules the quadrant" kind of thing. Until then, there are territories. Some are Fed territory, others are the corporate force territory. The two powers might have to do some negotiating if one has to traverse the other's territory.
However, even the corporate entity respects things like the Klingon and Romulan Neutral Zones. They have no desire to start a war, and understand that neither the Klingons, nor the Romulans are likely to discriminate between StarFleet forces and Corporate forces, especially if either tries to cross the Neutral Zones.
To the Federation's surprise, the corporate controlled worlds seem to prosper about as well as Federation planets. Little, if any conflict. No starvation. No faction wars. They aren't perfect utopias, but they do alright. Every now and again, the corporate forces might have to put out a small brush fire in their own territory, but nothing has broken out into full on open warfare.
This entity could be a dramatic foil for the Federation, without having to be overly antagonistic.
There might even come a time where the two forces realize that they can defend their respective territories as joint forces should the Klingons or Romulans decide to try an invasion, or start a war.