I believe Starfleet has always had a type of Military Force that was trained and equipped to address specific situations.
Such a force could simply be Starfleet's special forces (which do exist in novels). They don't need to be Marines.
Could be that Paramount know there were fan groups based on "Starfleet Marines" since the 1970's and did not want to use the same name for the M.A.C.O.s possible copy right issues and what not. Or even acknowledging the established fan groups already in existence.
So "the powers that be" for Star Trek decided instead to use the none-desrcript word "Military" in place of Marines.
The personal of the M.A.C.O.s used a Marine/Army rank structure that was different from Starfleet. The uniforms and weapons were different too and showed that they were trained to provide military support above Lt. Reeds ship security forces.
The MACOs were always meant to be a seperate organization from Starfleet, which is why they were called "MACOs" as opposed to "Starfleet Marines." There are no coypright issues since fan groups can't own the copyright to anything related to Star Trek. Besides, if Paramount did feature Starfleet Marines, and such a group wanted to take legal action against Paramount, such group could then potentially face legal action from the USMC on the same grounds.
No, the real reason the name MACOs was used as opposed to Marines was to establish that they were completely fictional and therefore they could get away without researching military practices and protocols without having to adhere to real life precedents.
In fact, in the original planning stages of Enterprise, Malcolm Reed was meant to be a "Starfleet Marine" with the rank of Major. Although a reason was never given for why they didn't go through with this, I would imagine it was to avoid complaints about not knowing what Marines really are.