The MACO's weren't a part of the Enterprise's command structure, so Reed could give orders to Hayes because Reed was part of Archer's command. Reed was acting using Archer's authority over Hayes Starfleet doesn't do things on stars, why do you think Marines only do things on water? My take is that given the importance of the mission Major Hayes replaced the platoon's usual lieutenant. There was a limit to how many MACO's could be added to the Enterprise's complement of people, so the lieuteneant was left on Earth.Khaki.
As far as Hayes v Reed, you're probably right. Although I also wonder if it would have better to have Reed as Acting XO (and 3-i-C) with the rank of Commander) and Trip as 4-i-C or 5-i-C) as either Commander or a Lieutenant. I certainly think that it would have allowed them to develop the character of Reed better and would have avoided the issue of Hayes outranking him.
As far as Marines go, obviously they don't
only do things on water, rapid deployment for landings from water (and to a lesser extent recon and intel-gathering) are certainly where they are mostly clearly
superior to other similar units (even then, only if you need large numbers of troops, SEALs and potentially USCG DOG are better for small-unit like VBSS work). However, for many of the functions that Starfleet might need - including but not limited to: unconvential warfare (Green Berets), airbourne assault (Army Rangers, AFSOCOM, FORSCOM) or artic/mountain warfare (10th Mountain)*, hostage rescue (Delta, HRT) or CBRNE (various including NEST and DOG) - are not typically as I understand it roles that align with the typical understanding of the Marine Corps' skillset.
* The only "on-screen" appearance of the 'Starfleet Marine Corps' (for Operation Retrieve) being an example of this, especially due the US-style format of the name (while the USMC certainly has mountain warfare capacity, it's not a focus IFAIK), whereas calling them the Federation Marines (or at a pinch the Starfleet Marines) would hint at a unit similar to the UK Royal Marines (of which 3 Commando, the main combat element are artic/mountain warfare experts and therefore the kind of unit you'd want to use on Rura Pente).
Note, this isn't my only issue which the Operation Retrieve scene. If West is supposed to be a member of the different Service (or even just a different Branch) then he should have a different shirt colour (possibly the dark blue of the "Federation forces" that assisted Kirk on Nimbus II in
TFF), ideally a different breast logo (if I was reworking the scene now, I'd probably base it around the AIG/Defense Branch (
USS Constellation) insignia from TrekLit), and also correct rank for his costume (him being refered to as Vice Admiral/Lieutenant General would probably be the easier "fix" and would make sense in context). The presence is also an unexplained oddity (though given their mutual associatons, it might have been at "West's request". However, my biggest issue with the Operation Retrieve scene is that they are using a
paper flip chart, something that was
vanishingly rare even in
TOS and looks even more ridiculously anachronistic after the fact, IMO?