Had the coup been successful, I suspect that either the Federation would fracture or there would a civil uprising to oust the military dictatorship. We don't know what Starfleet's membership is like, but there are billions of civilian inhabitants on over 150 member worlds as well as colonies, many of whom would object to the democratic process being usurped by Starfleet and would take action.