A military's primary purpose is to fight a war. That is not Starfleet's primary purpose. While Starfleet does indeed act as the de-facto defense force of the United Federation of Planets, its primary purpose is to explore space. Defense is an secondary, though still highly important, mission of Starfleet's. Because its primary purpose is not fighting war, it is not technically a military organization, which gels with statements made both on-screen, and in behind-the-scenes documentation, interviews, etc.
Actually, if you look at the countries that are regarded as "not having a military" this is isn't necessarily as wrong as the "Starfleet is a military" side of the arguement would have. Reading between the lines, it usually comes down to at least one of two or three factors: Do they have forces that are primarily intended for warfare (standing infantry, armor & artillery v armed police) and do they legally classify them as such. Japan's SDF satisfies only the second criteria, whereas Iceland - with an armed Coast Guard, an unarmed air force and no standing army - technically meets both. While Starfleet is a uniformed service organisation with militarised assets, we have no cannon evidence that dedicated warfare units exist (standing army, combat-only starship pre-USS Defiant), all examples of ground combat have been handled by security forces, and production documents (which may stand in place of in-universe law) suggest that Starfleet is not regarded as miitary by the rest of the Establishment even if a lot of their tradtions and competencies are traditionally associated with militaries.