The most interesting, truly "alien" alien race were the Neural parasites from "Operation--Annihilate!"
Mankind's longstanding fear of parasites/infection/loss of control is played to the bursting point with this episode's creatures. While they do look like a kind of jellyfish, they also take on the gruesome look of a large, open lesion: raw, torn-edged, yellowing, and sort of bloody in appearance.
Adding the fact the creatures were thinking
--not just some random beast operating from instinct, so their "alien" threat was magnified--so there's no saving grace to be found.
One can argue the creatures were more at home in 30's pulps or 50's horror films, but in the allegedly "better" world of 23rd century TOS, the Neural parasites were the inexplicable shock with no intent on peaceful co-existence, no easy answers.
Other favorites: The Tholian Assembly & the TOS Klingons, with the Tholians just as removed from a peaceful relationship / cultural exchange with the Federation. That is what you find out on the "final frontier."