That position-not-rank theory on dress uniforms is strangely alluring, for all its consistency. But I'd still go for it denoting exact ranks; the few mistakes might best be explained away, just like the reference to "LtCmdr Spock" is better explained away as a "dialogue oddity" rather than a "costuming error that lasts for exactly one episode".
As I recall, this is
not the only ep in which Spock is identified by spoken word as a
lieutenant commander while his sleeve braids apparently suggest he is actually a
full commander. I believe that Kirk does it again in the "little green men" encounter with Capt. Christopher in "Tomorrow is Yesterday".
On the trivia side, let's not forget that Mendez is never really associated with command of SB11. He merely happens to be there; his unusual uniform color choice might suggest command of a local fleet or flotilla, associating him more with the goldshirted Decker and Wesley than with the otherwise redshirted starbase staff and known starbase COs.
Commodore Stone could well have remained in charge - but he'd not be eager to converse with Kirk in the aftermath of the last embarrassment...
Timo Saloniemi
If you check out the scene at the end of "The Menagerie, Part II" when Uhura calls in from the bridge and reads a communique from "Mendez, J.I., Commodore, Starbase 11", you'll be as perplexed as I am about the whole thing.
Here's my theory:
Normal "duty fatigues" (the colored long-sleeve t-shirts and black trousers ensemble) are set up for everyday use. The color denotes which discipline the wearer is assigned to, the badge/patch denotes a unit/fleet (I like to assume that Kirk's arrowhead insignia represents the "First Fleet", or maybe some other regional organization in Federation space) and the symbol within the badge may indicate a department or perhaps a more specific assignment than the tunic color. And the sleeves indicate rank, although the braiding system does not make sense in TOS, and flatly contradict the spoken dialogue on more than one occasion.
Dress uniforms (or, at least we assume the shiny tunics we see are their dress uniforms, and not some less formal Class A's or something like that) may speak more to title ("I'm a CO", "I'm a department head") with their gold piping. The colored speckles on their chests may indicate ribbons and medals earned, or something else entirely (like rank, unit/flotilla, special titles, etc.). In this scenario, it is most interesting that rank is not necessarily spelled out (unless it is part of the colored speckles). And what suggests to me is that these are Class A's and not the most fancy version is that the shiny tunics still carry the discipline color. If this was like the U.S. Navy and they had an even high level of dress chokers, the uniforms might all be the same color (white).
As far as Mendez is concerned, you may right, Timo. Mendez could be Starfleet Command's senior administrator in Sector 11 (for sake of argument), while Stone may be busy running the sector's starbase. It could also be that Stone runs the base, the base is one of the Enterprise's "assigned ports" at that point, and Mendez commands all Federation starships (or is the field agent for an admiral who does) assigned to that port.
Does any of this make sense?