It's not that hard a concept to wrap one's head around. Starfleet handles the Federation's defense as part of its job but it does a lot more stuff that compromises about 60% of its job. The PD actually makes this possible, arguably. If the USA wasn't involved in foreign wars, we'll....the last war we'd have fought in was WW2.
For better or worse.
The issue is complicated by the fact that the show is ABOUT Starfleet so we never see Planetary Security for Earth. Also, the "defenses" Earth and other races have which get ignored by V'ger and other groups.