In my post number 155 on page 8 of this thread I point out that the Adnromeda Galaxy is not the closest galaxy to our Miky Way Galaxy.
https://www.trekbbs.com/threads/yet-another-doomsday-machine-thread.306036/page-8
See my anser here:
https://astronomy.stackexchange.com/questions/39879/what-is-the-closest-galaxy/39933#39933
And see the list here:
https://en.wikipedia.org/wiki/List_of_nearest_galaxies
For more evidence about knowledge of nearby galaxies among astronomers and the pubic in the 1960s, I found my copy of an astronomy textbook
Exploration of the Universe: Brief Edition, George Abell, 1964, 1969. Like most astronomy books it has tables, including a table of the nearest stars, and a table of the twenty brightest stars (as seen from Earth), etc. in the Appendixes. The table of the then 17 known members of the Local Group, is on page 402. In that table the Andromeda Galaxy is the second farthest galaxy in the Local group, second only to NGC 598 (M33). the Large and Small Magellanic Clouds are the first and second closest galaxies to our galaxy in that list.
The sentence that:
Altogether, there are 3 spirals, 4 irregulars, and 10 ellipicals, of which 6 are dwarf elliptical galaxies.
Shows that dwarf elliptical galaxies are counted as galaxies. On page 398 it is said:
Elliptical galaxies range all the way from the giants, just described, to dwarfs, which are believed to be the most common kind of galaxy. An example is the Leo II system, in figurge 21-11. There ae so few bright stars in this galaxy that even its central regions are transparent. The total number of stars, however (most too faint to show in Figure 21-11), is probably at least several million. The absolute magnitude of this typical dwarf is about - 10, it's luminosity is about 1 million times that of the sun.
If a galaxy as "small" as the Leo II dwarf galaxy is a typical dwarf galaxy, some dwarf galaxies could be a lot smaller than Leo II, and still count as galaxies.
And many other astronomy books from that era might have tables of the galaxies in the Local Group, or mention the Large Magellanic Cloud as the closest galaxy to our Mikly Way Galaxy.
The Guinness Book of Astronomy Facts & Feats, Pactrick Moore, 1979,1983, page 112, gives a similar but not identical list of 16 galaxies in the Local Group, inludingone, Maffei I, which is no longer considered part of the Local Group.
Considerng how large and conspicuous the Large Magellanic Cloud looks from Earth, it should have been one of the first external galaxies to have its distance measured in the 1920s, and so should have been recognized as an external galaxy much closer than the Andromeda Galaxy since before 1930.
The term "The Local Group" was introduced by
Edwin Hubble in Chapter VI of his 1936 book
The Realm of the Nebulae.
[8] There, he described it as "a typical small group of nebulae which is isolated in the general field" and delineated, by decreasing luminosity, its members to be
M31,
Milky Way,
M33,
Large Magellanic Cloud,
Small Magellanic Cloud,
M32,
NGC 205,
NGC 6822,
NGC 185,
IC 1613 and
NGC 147. He also identified
IC 10 as a possible part of the Local Group.
https://en.wikipedia.org/wiki/Local_Group#History
In the climax of James Blish's novel
Earthman, Come Home (1955) New York City travels to the Large Magellanic Cloud, clearly described as a different galaxy.
http://www.isfdb.org/cgi-bin/title.cgi?2026 This section is based on earlier stories, "Sargasso of Lost Cities" and "Earthman, Come Home", both published in 1953. It is quite possible that the distance to the Large Magellanic Cloud is mentioned in
Earthman, Come Home (1955) and/or its sequel,
The Triumph of Time (1958)
http://www.isfdb.org/cgi-bin/title.cgi?961
Since the Andromeda Galaxy is rather similar in size to our Milky Way Galaxy, being a bit larger, the Milky Way Galaxy should look with unaided vision just as small and dim from the Andromdea Galaxy as the Andromeda Galaxy looks from Earth in the Milky Way Galaxy, a tiny smudge of dim light just barely visible to the unaided eye. But if I remember correctly both
Earthman, Come Home (1955) and
The Triumph of Time (1958) mention how large and spectacular the Milky Way Galaxy looks from the Large Magellanic Cloud, thus showing that the Large Magellanic Cloud is much closer to Earth and the Milky Way Galaxy than the Andomeda Galaxy is.
Similarly the climax of Robert A. Heinlein's novel
Have Spacesuit--Will Travel (1958) is set in the Small Magellanic Cloud and the spectacular view of our Milky Way Galaxy is mentioned, thus showing that the Small Magellanic Cloud has to be much closer to our galaxy than the Andromeda Galaxy is.
A.E. Van Vogt's novel
The Mixed Men (1952)
http://www.isfdb.org/cgi-bin/title.cgi?22964 involves an Earth ship (which makes the Enterprise seem as primitive as a Project Mercury capsule) finding a secret colony of Earth people in the Large Magellanic Cloud with lots of political and military intrigue. It is a combination of shorter stories first published in 1943 to 1945. And it is likely that the distance to the Large Magellanic Cloud may be mentioned, or the Large Magellanic Cloud might be described as the nearest galaxy to the Milky Way.
So it was certainly possible when TOS was being made for amateur astronomers or science ficiton fans to know that there are galaxies closer to the Milky Way than the Andomeda Galaxy is.