I think you've gotten a few things confused. You might be thinking of a proposed animated Trek series set in the 25th century which was never greenlit. It dealt with Vulcan, and other UFP worlds, seceding from the Federation after several Omega molecules had been released, rendering interstellar travel next to impossible in vast areas.
You might also be thinking of the fall of Betazed in the the sixth season of DS9. The powers that be wanted the Dominion to conquer a planet that meant something to viewers, so as to give Sisko a justifiable reason for what he does in In The Pale Moonlight. Originally the powers that be thought about having Vulcan be the planet conquered by the Dominion. However, it was decided that Vulcan carried TOO much weight with the viewers, so it was changed to Betazed.
As far as I know, DS9 never considered having Vulcan leave the Federation.