I have never had the opportunity to watch TOS and I definitely plan to get it on DVD. It's insane IMO to consider something that started ST phenomenom not a part of it. Of course it might have its mistakes and things, characters or situations that seem ridiculous now, but still it's how it all began and as such should be respected and a part of canon.
Well, there are certain foundational material facts which get repeated (many others get contradicted) throughout the series which are, no doubt, canonical.
Look at it this way:
The Star Wars saga officially begins with Episode I and Jar Jar. For many children, Star Wars begins (experientially/chronologically) with episode I. Films 4-6 (however vaguely) refer to events which occurred in films 1-3.
And yet many Star Wars fans hold that 1-3 aren't really part of SW canon. No in denying that 1-3 are canon are they denying the references in films 4-6? No, they are simply denying that those references refer to what they saw in films 1-3. They would rather keep the references vague and undefined.
You could dump TOS as canon without denying canonical status of the references we find in the later films. When we think of the original Trek era, we should simply have a vague sense of it existing like a mystery, like the "clone wars" when we originally saw Star Wars.
The ghost vaguely appears in ENT's Mirror Universe episode and in the Tribble episode on DS9, and a few guest appearances, but a ghost it remains - the bad sfx, the Kirk-fu, the chauvinism remains discretely in the background.
At least, that is a likely sort of counter-argument you'll encounter.
Is the only way to stop this madness for us to stop refereeing what is really Trek?