Starfleet has the rank of Commodore- above a Captain but below a Rear Admiral Lower Half.
Umm, as of today, Commodore is the same thing as Rear Admiral Lower Half (RDML). Or at least the last time Commodore was used in the USN, it was just a different name for RDML. (There are several other meanings for the word Commodore, too, as you say.) The Wikipedia has the dirt on this; basically, Commodore has been an actual rank grade every now and then, a temporary title at other times, and there has always been a bit of congestion at the lowest flag ranks or highest line ranks, with the navies trying to cheapskate their way through employing these experienced senior officers even when there aren't enough Admiralty jobs for them. Usually, hilarity and confusion has ensued.
The last time Commodore was made an actual grade in the USN, in the 1980s, people with said rank basically raised a stink because their title didn't contain the word "Admiral" - so "Rear Admiral, Lower Half" (RDML) was chosen as the more palatable name, to fit between Captain and Rear Admiral (RADM).
Star Trek has never been quite as messy about the issue: in TOS, Commodores were obvious flag officers with an easily identifiable rank braid of their own, and were seen commanding both starbases and ships or formations of ships, as well as handling assorted desk jobs. The rank has not been mentioned much in later Trek, and it could be argued it has been renamed or removed, but there's no solid proof on that.
Whether the United Earth Starfleet in ENT had the rank of Commodore is not in question, either. Max Forrest explicitly held that rank in the flashback episode "First Flight".
Reed was a Lieutenant but he outranked Major Hayes?!?
Well, Hayes was but a guest aboard the ship. The lowest ablespaceman would probably have outranked the Major in certain issues concerning the ship... Only the internal security tasks of the ship would have been a bit of a grey area, and there Archer might have made the useful clarification that Reed would be in charge and Hayes would only command the MACOs in landing or boarding missions.
Apparently, the use of units from the Military Assault Command aboard ships of Starfleet Command was a novel or rare practice during the third season of ENT. Various special arrangements would then only be natural; also, Archer would have personal leeway in organizing his ship, and in reorganizing and re-reorganizing it as best befitted the important mission where he couldn't contact his own superiors for permission.