Good thread! One of the things I've liked in Star Trek TOS from the beginning was quantity of horror obertones, weirdly but well combined with optimism and brightness of the whole show. The first episode I saw -
The Man Trap - is the closest to horror, in my opinion. Good creepy atmosphere and really scary, despite of being made of something like fishnet, monster.
That Which Servives seems more the combination of horror and beauty, but it's a good example too. It worth noting too great amount of exellent black humour here, nice addition to the tension of the whole episode. Other frightening eps are two "Children-of-the-Corn-episodes" -
Miri and
And the Children Shall Lead. The first has a lot of interesting features besides the horror, but only good creepiness saves the other (and only for a horror fan, I suppose).
I think
The Return of the Archons is the greatest example. It's more dystopia in fact, but it seems the best horror TOS ep for me and one of the best at all. Brainwashed society with mad leader, misterious "Red Hour", taste of the turn of XIX-XX centuries - times, good for horror (after this ep Night Shyamalan's film
Village seems less original for me), sinister music, ominous, a bit Lovecraftian atmosphere, - of the most optimistic books by Lovecraft, of course.
I can remember too three episodes with scripts by Robert Bloch -
What Are Little Girls Made Of, Catspaw and
Wolf in the Fold. They aren't so scary but worthy mention too. I hadn't respected Bloch so much before I saw them... but references of the Great Old Ones! The whole Halloween-themed episode with two little Cthulhus at the end! Jack the Ripper! Jack the Ripper in Star Trek!

I know how much Bloch adored him.
Besides, there are a lot of another good examples that frighten in their own way. Two with the theme of madness -
Dagger of the Mind and
Whom Gods Destroy, some parts of
The Devil in the Dark, Spectre of the Gun and
Charlie X, Operation - Annihilate! (the Puppet's Musters),
The Deadly Years, Obsession, The Lights of Zetar (more exorcism!),
The Tholian Web and two scary in more non-horror way -
The Empath and
Plato's Stepchildren.