My theory on Marines in Starfleet is that there's hundreds of security personnel on Federation starships out of a crew of a thousand so there's no real point to carrying around a couple of hundred extra marines for repelling boarding invasion (which the security personnel seem to do when not carrying out police duties).
Likewise, if they DID EXIST, they'd be the ones who would be going onto hostile starships with Away teams since that would be 90% of what a Marine would do if they existed in space (see Star Wars Stormtroopers). The fact there's no sign of Marines in Star Trek says to me they don't exist anywhere but the Enterprise era, probably.
Nevertheless, I think they probably exist on a planetary level as mentioned above. Starfleet isn't a standing army but I wouldn't be surprised if planets in the Federation maintain defense forces. As for how to handle land invasion? That's an interesting question and may well be an irrelevant question of war in the 24th century.
Honestly, this isn't a slight against Star Trek security, I think we're meant to assume they're pretty badass--we just never get to see it.