All of which is irrelevant to the question of whether Starfleet could enforce a policy of non-genocide. Yeah, it could, even when its starships come under attack - it would just call for the skippers to have the courage to withdraw or to commit suicide.
The particulars of the policy would be a separate issue. Do only nicely behaving sentients deserve protection? Today, we don't think that way - we allow deadly, senselessly man-eating predators to multiply and take back lands that at one point were tamed by man. We can afford to; it's difficult to see why the Federation could not as well.
Absolutes never make for good policy, though. The Federation has shown that it is willing to erase species if those pose too big a threat to too big or important a part of the UFP. It has also claimed it doesn't want to expand at excess cost to others, although the exact cost varies from case to case. In cases like Vaal or Landru, no Federation interests were in danger, other than the throwaway lives of Kirk and friends, or further trespassers of their ilk. In cases like the Eminians, though, there was a threat or fear thereof that failure to intervene would hurt UFP assets other than trespassing starship crews. And in case of the Space Amoeba, the threat was clear and present and
modified by the fact that the enemy was a vile nonhuman beast evoking no sympathies.