It's pretty clear that in the earliest episodes of TOS, the United Federation of Planets, as Trek fans know it today, hadn't been conceived of yet. Kirk mentions that he reports to the United Earth Space Probe Agency (UESPA) in Charlie X and Tomorrow is Yesterday, and likewise refers to the Enterprise as a United Earth Ship in The Corbomite Maneuver. Similar ideas are present in other early episodes--no mention of a "Federation" occurs, but references to "Earth vessels," "Earth outposts," "Earth colonies," and "Earth Science" abound. And of course, the script for The Cage says that Spock is "the only exception to the familiar types represented by the crew," i.e. the only alien on board.
So when did the Federation as ultimately conceived (a multi-species, multi-member interstellar government with diverse starship crews) take root?
The first time "the Federation" was mentioned in TOS was in Arena. Yet that very episode also refers to a destroyed "Earth observation outpost" on Cestus III.
A Taste of Armageddon was the first episode to name "the United Federation of Planets." However, while the people of Eminiar VII talk about their "Federation prisoners," they also refer to the "Earth party," "the Earth ship," and "the Earth Ambassador." In a similar vein, Spock describes the USS Valiant, destroyed 50 years ago, as leading an "Earth expedition." (Interestingly, the "Earth ambassador" in this episode is seeking not to have Eminiar VII join the Federation, but rather to negotiate the creation of a treaty port--similar to British imperialist treaty ports like Hong Kong).
If one takes these references on their own, without referring to later episodes, it seems the "United Federation of Planets" may originally have been intended as a "United Federation of (Terran) Planets." That is, the Federation as first conceived would have been simply the united government of Earth and its planetary colonies.
This idea has one very important precedent. In Forbidden Planet, the United Planets was the governing organization which sent out the crew of the C-57D. Notably, as the shooting script and deleted scenes show, the humans of Forbidden Planet have not yet made first contact with any alien race. Thus, the United Planets is an all-human government, administering Earth and its colony worlds. (And as those same deleted scenes reveal, an early version of the film's opening narration mentioned that humanity was "now banded together in a single federation." )
Since Forbidden Planet inspired so much of early Star Trek, I strongly suspect it also provided the origin of Gene Roddenberry's concept of the "United Federation of Planets." In which case, it would be natural to envision the UFP as chiefly a government of Terran worlds.
Notably, although Roddenberry's March 1964 Star Trek pitch refers to the SS Yorktown as a United Space Ship sent out from Earth, which would aid "Earth colonies" and investigate threats to "Earth security," the document also mentions that the Yorktown would regulate the activities of any "Federated commerce vessels and traders" it encountered. Since these words were written well before the creation of UESPA and the "Earth ship" Enterprise, it seems likely that the "Federation" of the 1964 pitch was a Human Federation.
The first mention of species besides Humans being part of the Federation occurs in Errand of Mercy, in which both Kirk and the Klingons know that "Vulcan is a member of the Federation." Yet, rather infamously, in a later episode, Friday's Child, Kirk refers to "the Earth Federation."
The plot of Friday's Child involves treaty negotiations for Federation mining rights on the planet Capella--NOT negotiations for Capella's entry into the Federation. As Kirk tells the Capellans, "your world is yours and will always remain yours." When taken together with A Taste of Armageddon, the Federation of these early episodes seems to be slow (and not especially eager) to admit new members.
I hypothesize that, at this stage (late Season 1/early Season 2) of the writing of TOS, the "Earth Federation" was still seen as primarily a government of Earth and its colonies. Thus, alien species such as Vulcans could join as members, but only if they wished to--otherwise, the Federation would generally be content with treaties to secure ports and mining rights on friendly alien worlds. In short, the Federation would be made up of mainly human worlds, along with a few alien planets that have decided to join, while other planets would be linked to it solely by commercial treaties.
I'd say that Journey to Babel, in which we see lots of alien Federation ambassadors aboard the Enterprise, really marks the beginning of the shift toward the multi-species Federation of the movies and TNG. (IIRC, this is also the first episode in which we see a planet petition to enter the Federation.)
But even then, we still don't have multi-species starship crews: the Intrepid in The Immunity Syndrome is crewed by 400 Vulcans.
Such a system isn't unprecented in pulp SF literature. Edmond Hamilton was the first major SF author to write about a multi-species interstellar Federation, back in the early 1930s. In his first such story, The Star-Stealers, it's mentioned that each member world of the Federation of Suns contributes a quota of ships and crews to the fleet of the Interstellar Patrol, so that a given ship's crew is all-human or all-Canopan, all-Rigellian, etc. (Hamilton's later stories about the Federation of Suns did show multi-species starship crews, much as the ST movies and TNG included more aliens serving side-by-side with humans.)
A quota system also makes sense for the TOS Starfleet, if we imagine that each member planet of the Federation provides a certain number of starship crews. Thus, crews would be all-Human or all-Vulcan, etc. (with the occasional exception--but then again, Spock was the only alien we saw on the Enterprise crew in TOS). This would allow the species of each ship to set the environmental controls and artificial gravity at levels comfortable to them.
Multi-species starship crews were first anticipated in 1968's The Making of Star Trek, which asserted that although starship crews were always primarily of one species, other species were represented aboard the ships in lesser numbers. Thus, although Spock was the only Vulcan aboard the Enterprise, he was not the only alien. Obviously, we never got to see this idea in action on TOS, but TAS (with Arex and M'Ress) and Phase II/TMP (with Ilia) ran with it.
To sum up: it would seem that the Federation of modern Star Trek was not achieved all at once, but rather in a series of gradual evolutionary steps over the course of TOS. When first mentioned on screen, the UFP was simply UESPA with a different name, and it only slowly broke away from that idea.
So when did the Federation as ultimately conceived (a multi-species, multi-member interstellar government with diverse starship crews) take root?
The first time "the Federation" was mentioned in TOS was in Arena. Yet that very episode also refers to a destroyed "Earth observation outpost" on Cestus III.
A Taste of Armageddon was the first episode to name "the United Federation of Planets." However, while the people of Eminiar VII talk about their "Federation prisoners," they also refer to the "Earth party," "the Earth ship," and "the Earth Ambassador." In a similar vein, Spock describes the USS Valiant, destroyed 50 years ago, as leading an "Earth expedition." (Interestingly, the "Earth ambassador" in this episode is seeking not to have Eminiar VII join the Federation, but rather to negotiate the creation of a treaty port--similar to British imperialist treaty ports like Hong Kong).
If one takes these references on their own, without referring to later episodes, it seems the "United Federation of Planets" may originally have been intended as a "United Federation of (Terran) Planets." That is, the Federation as first conceived would have been simply the united government of Earth and its planetary colonies.
This idea has one very important precedent. In Forbidden Planet, the United Planets was the governing organization which sent out the crew of the C-57D. Notably, as the shooting script and deleted scenes show, the humans of Forbidden Planet have not yet made first contact with any alien race. Thus, the United Planets is an all-human government, administering Earth and its colony worlds. (And as those same deleted scenes reveal, an early version of the film's opening narration mentioned that humanity was "now banded together in a single federation." )
Since Forbidden Planet inspired so much of early Star Trek, I strongly suspect it also provided the origin of Gene Roddenberry's concept of the "United Federation of Planets." In which case, it would be natural to envision the UFP as chiefly a government of Terran worlds.
Notably, although Roddenberry's March 1964 Star Trek pitch refers to the SS Yorktown as a United Space Ship sent out from Earth, which would aid "Earth colonies" and investigate threats to "Earth security," the document also mentions that the Yorktown would regulate the activities of any "Federated commerce vessels and traders" it encountered. Since these words were written well before the creation of UESPA and the "Earth ship" Enterprise, it seems likely that the "Federation" of the 1964 pitch was a Human Federation.
The first mention of species besides Humans being part of the Federation occurs in Errand of Mercy, in which both Kirk and the Klingons know that "Vulcan is a member of the Federation." Yet, rather infamously, in a later episode, Friday's Child, Kirk refers to "the Earth Federation."
The plot of Friday's Child involves treaty negotiations for Federation mining rights on the planet Capella--NOT negotiations for Capella's entry into the Federation. As Kirk tells the Capellans, "your world is yours and will always remain yours." When taken together with A Taste of Armageddon, the Federation of these early episodes seems to be slow (and not especially eager) to admit new members.
I hypothesize that, at this stage (late Season 1/early Season 2) of the writing of TOS, the "Earth Federation" was still seen as primarily a government of Earth and its colonies. Thus, alien species such as Vulcans could join as members, but only if they wished to--otherwise, the Federation would generally be content with treaties to secure ports and mining rights on friendly alien worlds. In short, the Federation would be made up of mainly human worlds, along with a few alien planets that have decided to join, while other planets would be linked to it solely by commercial treaties.
I'd say that Journey to Babel, in which we see lots of alien Federation ambassadors aboard the Enterprise, really marks the beginning of the shift toward the multi-species Federation of the movies and TNG. (IIRC, this is also the first episode in which we see a planet petition to enter the Federation.)
But even then, we still don't have multi-species starship crews: the Intrepid in The Immunity Syndrome is crewed by 400 Vulcans.
Such a system isn't unprecented in pulp SF literature. Edmond Hamilton was the first major SF author to write about a multi-species interstellar Federation, back in the early 1930s. In his first such story, The Star-Stealers, it's mentioned that each member world of the Federation of Suns contributes a quota of ships and crews to the fleet of the Interstellar Patrol, so that a given ship's crew is all-human or all-Canopan, all-Rigellian, etc. (Hamilton's later stories about the Federation of Suns did show multi-species starship crews, much as the ST movies and TNG included more aliens serving side-by-side with humans.)
A quota system also makes sense for the TOS Starfleet, if we imagine that each member planet of the Federation provides a certain number of starship crews. Thus, crews would be all-Human or all-Vulcan, etc. (with the occasional exception--but then again, Spock was the only alien we saw on the Enterprise crew in TOS). This would allow the species of each ship to set the environmental controls and artificial gravity at levels comfortable to them.
Multi-species starship crews were first anticipated in 1968's The Making of Star Trek, which asserted that although starship crews were always primarily of one species, other species were represented aboard the ships in lesser numbers. Thus, although Spock was the only Vulcan aboard the Enterprise, he was not the only alien. Obviously, we never got to see this idea in action on TOS, but TAS (with Arex and M'Ress) and Phase II/TMP (with Ilia) ran with it.
To sum up: it would seem that the Federation of modern Star Trek was not achieved all at once, but rather in a series of gradual evolutionary steps over the course of TOS. When first mentioned on screen, the UFP was simply UESPA with a different name, and it only slowly broke away from that idea.