The ultimate episode ranking thread

[Captrek]

Those pics in the previous post are clickable thumbnails, in case that wasn’t clear.

Anybody care to opine as to why this thread was such a spectacular failure, when other episode-rating threads get much more participation?

Is it a marketing failure?

Is rating all 79 episodes too much work? You only have to rate the ones you want to rate, and with the truncate feature you can just name your favorites like in a top 10 thread.

 

[CorporalCaptain]

I snarfed the pics. I've studied the ranking, and it's really interesting.

Yes, it sounds like a lot of work. If I'm a typical geek, then I suspect geeks will tend to want to do this sort of thing perfectly or not at all.

I probably ended up spending about 30 minutes on the ranking, not all in one sitting. First I sorted the episodes into quartiles, then I ranked them within each quartile. I only finished when I overcame my instinct not to do a perfect ranking, so as not to leave you hanging (but I think I did it pretty accurately, actually).

For me, developing a perfect ranking would require considering all 79*78/2=3081 comparisons to decide which episode is better in each pair of episodes. To look at it another way, that's answering 39 times as many questions as assigning each episode a number. Not only that, that relation would have to be checked to make sure it's acyclic, more work, and only then topologically sorted. I seriously doubt everyone would instinctively want to do that, but I suspect that's nevertheless indicative of just how much work could be involved, either explicitly or implicitly.

 

[Isis]

...Anybody care to opine as to why this thread was such a spectacular failure, when other episode-rating threads get much more participation?...

Maybe it's an idea whose time hasn't come yet.

That said, I don't know why the initial response to this thread came from just a few posters. To me, it's an interesting idea, and if one does a "quick and dirty" rating like I did, the time investment is minimal. (And by "quick and dirty," I mean just quickly give your first impression for each rating.) I wish I could offer something more constructive, but I can't at this point.

Finally, on the diagrams themselves, I'll note how the shows seem to be "congregating together" by color (with yellow, blue, and red representing Seasons 1, 2, and 3, respectively?). Then again, with the results of only four people so far, maybe it's too early to make any generalizations.

In any case, captrek, at least all the cool people voted.

 

[Captrek]

Isis' Evil Twin: Yes, the colors represent seasons. I put them in there to make the graphs more visually striking, but as you noticed it also tells a sad story of declining quality from season to season (at least in the minds of these four voters).

Another thing that stands out to me is how “clean” the top of the graph is. The top 23 episodes (from The City on the Edge of Forever and Balance of Terror through This Side of Paradise) make a simple, very legible subgraph. Below that, it gets messy.


CarbonCopy: You present an interesting hypothesis, that the enormous expressive power of the ballot chased off a board full of perfectionists who would rather not participate at all than participate without using all that expressive power. However, my impression is that we do not have a board populated entirely or almost entirely by perfectionists, so I doubt the hypothesis.

I think it’s a marketing failure.

I think the ballot may have intimidated some people. I intended for it to be very simple, but maybe it’s not as simple as I hoped, or I failed to effectively convey its simplicity.


If I were to do it over again, I would emphasize up front that the requirements for participation are almost nil. It’s just an invitation to say whatever you care to say about whatever episodes you care to say something about. If you want to rate all the episodes, fine. If you want to rate some of them, fine. If you just want to list your favorites, fine.

I would not require participants to use a properly formatted ballot. It makes my job a little easier when they do, but if somebody simply posts a top-ten list or whatever, I can convert it to a properly formatted ballot. It doesn’t take much time. I, for one, find the ballot format to be very intuitive and easy to use, but then I’m the one who defined it, so I would, wouldn’t I?

I would put a bunch of exclamation marks in the thread title. I don’t know why, but it seems to help.

 

[CorporalCaptain]

Another thing that stands out to me is how “clean” the top of the graph is. The top 23 episodes (from The City on the Edge of Forever and Balance of Terror through This Side of Paradise) make a simple, very legible subgraph. Below that, it gets messy.

Yes, this is interesting.

my impression is that we do not have a board populated entirely or almost entirely by perfectionists

This is probably a good thing. We'd be at each others' throats even more!

if somebody simply posts a top-ten list or whatever, I can convert it to a properly formatted ballot

Yes. So, unless the user said they were in order, you would assign the ten episodes listed a 2 and every other episode a 1, right? Whereas, otherwise of course you'd assign them 11 down to 2, respectively and as appropriate, and every other episode a 1?

You know, for shits and giggles, you could just peruse the top ten threads, format ballots yourself from those posts, and then post the results here.

 

[Captrek]

You know, for shits and giggles, you could just peruse the top ten threads, format ballots yourself from those posts, and then post the results here.

I like that idea. Good suggestion.

I’ve made ballots for the posts in [THREAD=147974]these[/THREAD] [THREAD=148915]two[/THREAD] threads. I replaced the “truncate” feature (which nobody had used) with an “all others” feature to accommodate the “sandwich” thread.

KirksStuntMan was a little difficult to deal with because he posted in both threads and his two posts were not consistent. I took my best stab at a composite.

Here are the ballots from those two threads.


Admiral Shran:

100  Journey to Babel
 90  The Conscience of the King
 80  Amok Time
 70  The Devil in the Dark
 60  Arena
 50  The City on the Edge of Forever
 40  The Enterprise Incident
 30  Balance of Terror
 20  Space Seed
 10  The Doomsday Machine
  0  all others

BillJ:

 1  A Taste of Armageddon
 1  The Omega Glory
 1  All Our Yesterdays
 1  Where No Man Has Gone Before
 1  The Doomsday Machine
 1  The Ultimate Computer
 1  Dagger of the Mind
 1  This Side of Paradise
 1  A Private Little War
 0  all others

Desmondidae:

 1  The City on the Edge of Forever
 1  Space Seed
 1  The Trouble with Tribbles
 1  Mirror, Mirror
 1  The Doomsday Machine
 1  The Devil in the Dark
 1  The Enterprise Incident
 1  Journey to Babel
 1  Amok Time
 1  The Ultimate Computer
 1  Dagger of the Mind
 1  The Tholian Web
 1  Balance of Terror
 1  A Taste of Armageddon
 0  all others
-1  Spock's Brain
-1  The Way to Eden
-1  The Omega Glory
-1  And the Children Shall Lead
-1  That Which Survives
-1  The Alternative Factor
-1  The Gamesters of Triskelion
-1  The Mark of Gideon
-1  Miri
-1  The Lights of Zetar
-1  Plato's Stepchildren
-1  Let That Be Your Last Battlefield

Dorian Thompson:

 1  The Doomsday Machine
 1  Journey to Babel
 1  Balance of Terror
 1  The Empath
 1  The City on the Edge of Forever
 1  The Trouble with Tribbles
 1  The Tholian Web
 1  Mirror, Mirror
 1  Charlie X
 1  The Naked Time
 1  Obsession
 1  Amok Time
 1  The Galileo Seven
 0  all others
-1  And the Children Shall Lead
-1  The Alternative Factor
-1  The Way to Eden
-1  The Mark of Gideon
-1  For the World is Hollow and I Have Touched the Sky
-1  The Gamesters of Triskelion
-1  Dagger of the Mind
-1  The Squire of Gothos
-1  The Conscience of the King
-1  Spock's Brain

KirksStuntMan:

100  Mirror, Mirror
 90  The Doomsday Machine
 80  Shore Leave
 70  Elaan of Troyius
 60  Metamorphosis
 50  Requiem for Methuselah
 40  The Tholian Web
 30  The Empath
 10  Obsession
  2  The City on the Edge of Forever
  1  What Are Little Girls Made Of?
  0  all others
 -1  And the Children Shall Lead
 -1  Who Mourns for Adonais?
 -1  Wolf in the Fold
 -1  The Apple
 -1  I, Mudd
 -1  Bread and Circuses
 -1  A Private Little War
 -1  A Piece of the Action
 -1  Patterns of Force
 -1  Plato's Stepchildren

Space Therapist:

 1  The City on the Edge of Forever
 1  Space Seed
 1  The Trouble with Tribbles
 1  Mirror, Mirror
 1  The Doomsday Machine
 1  Journey to Babel
 1  Amok Time
 1  The Immunity Syndrome
 1  Balance of Terror
 1  The Way to Eden
 1  The Lights of Zetar
 1  Patterns of Force
 1  Tomorrow is Yesterday
 1  All Our Yesterdays
 1  The Deadly Years
 0  all others
-1  Return to Tomorrow
-1  Wolf in the Fold
-1  Let That Be Your Last Battlefield
-1  That Which Survives
-1  The Mark of Gideon
-1  Miri
-1  Catspaw
-1  Charlie X
-1  The Squire of Gothos
-1  The Gamesters of Triskelion
-1  Wink of an Eye
-1  Turnabout Intruder
-1  Plato's Stepchildren
-1  And the Children Shall Lead

Vasquez Rots:

100  Amok Time
 90  Bread and Circuses
 80  The Conscience of the King
 70  The City on the Edge of Forever
 60  A Taste of Armageddon
 50  Journey to Babel
 40  The Corbomite Maneuver
 30  Errand of Mercy
 20  Who Mourns for Adonais?
 10  The Paradise Syndrome
  0  all others

VonFrank:

100  The Doomsday Machine
 90  The Ultimate Computer
 80  The Trouble with Tribbles
 70  The Enterprise Incident
 60  Balance of Terror
 50  Assignment: Earth
 40  The Tholian Web
 30  Elaan of Troyius
 20  Space Seed
 10  Tomorrow is Yesterday
  0  all others

Windows_Me_Guy:

100  Metamorphosis
 90  The Conscience of the King
 80  This Side of Paradise
 70  Mirror, Mirror
 60  The Doomsday Machine
 50  Bread and Circuses
 40  The Paradise Syndrome
 30  Spectre of the Gun
 20  The Man Trap
 10  That Which Survives
  0  all others

cmlmc1296:

100  The Doomsday Machine
 90  Mirror, Mirror
 80  The Enterprise Incident
 70  Balance of Terror
 60  Shore Leave
 50  The City on the Edge of Forever
 40  The Trouble with Tribbles
 30  Journey to Babel
 20  The Ultimate Computer
 10  The Tholian Web
  0  all others

plynch:

 1  The Devil in the Dark
 1  The Doomsday Machine
 1  Is There in Truth No Beauty?
 1  A Taste of Armageddon
 1  Amok Time
 1  Spectre of the Gun
 1  The Way to Eden
 1  The Conscience of the King
 1  Errand of Mercy
 0  all others
-1  That Which Survives
-1  Mudd's Women
-1  I, Mudd
-1  A Piece of the Action
-1  The Immunity Syndrome
-1  Catspaw

roger1999:

90  The City on the Edge of Forever
80  The Menagerie, Part I
80  The Menagerie, Part II
70  The Naked Time
60  Balance of Terror
50  The Empath
40  This Side of Paradise
30  Shore Leave
20  Journey to Babel
10  Mirror, Mirror
 0  all others

 

[Captrek]

I mentioned earlier that the graph was getting less tangled with each additional ballot. Sadly, that pattern has ceased to hold. After 16 ballots, the graph is as tangled as it was after 4.

The 12 new ballots are from a thread for listing your favorites and a thread for listing your favorites and least favorites. I probably shouldn’t be surprised that they have done little to clarify the picture with regard to middle-quality episodes, which is where most of the tangling is. Plus the fact that we still have nearly five times as many candidates as voters.

Here are the results after 16 ballots:

 

[CorporalCaptain]

That's totally awesome. It's totally untangled at the top and bottom.

 

[Gojira]

Here is my revised list. Best is a score of 5 and the worst is a 0.

5	The Cage	 
4	 Where No Man Has Gone Before	 	
5	 The Corbomite Maneuver	 
3	 Mudd's Women	 
3	 The Enemy Within	 	
3	 The Man Trap	  
4	 The Naked Time	  
0	 Charlie X	 	
5	 Balance of Terror	 	 
2	 What are Little Girls Made of?	 	 
3	 Dagger of the Mind	 	 
1	 Miri	 	
1	 The Conscience of the King	 	 
4	 The Galileo Seven	 	 
4	 Court Martial	 	 
5	 The Menagerie, Part One		 
5	 The Menagerie, Part Two	
4	 Shore Leave	 12/29/66	
0	 The Squire of Gothos	 	
5	 Arena	 	 
0	 The Alternative Factor	  
5	 Tomorrow is Yesterday		 
4	 The Return of the Archons	 	 
5	 A Taste of Armageddon	 	
5	 Space Seed		 
5	 This Side of Paradise	 	 
5	 The Devil in the Dark	 	 
5	 Errand of Mercy	 	
5	 The City on the Edge of Forever		
3	 Operation: Annihilate!

Season 2.

0	 Catspaw	 
5	 Metamorphosis	 	 
4	 Friday´s Child	 	 
4	 Who Mourns for Adonais?	 	 
5	 Amok Time	 	 
5	 The Doomsday Machine	 
0	 Wolf in the Fold		 
5	 The Changeling	 	 
4	 The Apple	 	 
5	 Mirror, Mirror	 	
5	 The Deadly Years	 	 
3	 I, Mudd		 
5	 The Trouble With Tribbles	 
3	 Bread and Circuses		 
5	 Journey to Babel	 	 
3	 A Private Little War	 	 
2	 The Gamesters of Triskelion	 	 
1	 Obsession	 	
5	 The Immunity Syndrome		 
4	 A Piece of the Action	 	 
4	 By Any Other Name		 
2	 Return to Tomorrow	 	 
5	 Patterns of Force	  
5	 The Ultimate Computer	 	 
3	 The Omega Glory	 	 
4	 Assignment: Earth		 

Season 3
	
4	 Spectre of the Gun	 	 
4	 Elaan of Troyius	 	 
5	 The Paradise Syndrome	 	
5	 The Enterprise Incident	 	 
0	 And the Children Shall Lead	 	 
4	 Spock's Brain	 	 
3	 Is There in Truth no Beauty?	 	 
3	 The Empath	 	
5	 The Tholian Web	 	 
4	 For the World Is Hollow and I Have Touched the Sky	 	 
4	 Day of the Dove	 	 
0	 Plato´s Stepchildren	 	 
1	 Wink of an Eye	 	 
0	 That Which Survives	 	 
3	 Let That Be Your Last Battlefield	 	
4	 Whom Gods Destroy	 	 
2	 The Mark of Gideon	 	 
5	 The Lights of Zetar	 	 
4	 The Cloud Minders	 	 
5	 The Way to Eden	 	 
5	 Requiem for Methuselah	 	 
4	 The Savage Curtain	 	
5	 All Our Yesterdays	 	 
1	 Turnabout Intruder

 

[Captrek]

That's totally awesome. It's totally untangled at the top and bottom.

You noted that 79 candidates have 3081 pairwise contests. I can’t reasonably list them all. However, the top 8 have only 28 pairwise contests among them. Here’s how they work out (not including Space Therapist’s revised list):

The Doomsday Machine:

WINS 6- 4 against The City on the Edge of Forever
WINS 6- 3 against Balance of Terror
WINS 7- 4 against Journey to Babel
WINS 6- 3 against Mirror, Mirror
WINS 7- 2 against Amok Time
WINS 10- 1 against The Trouble with Tribbles
WINS 10- 2 against The Devil in the Dark​

The numbers don’t add up to 16 because not every voter expresses a preference between every pair of episodes.
The City on the Edge of Forever:

WINS 4-2 against Balance of Terror
WINS 5-1 against Journey to Babel
WINS 5-3 against Mirror, Mirror
WINS 5-3 against Amok Time
WINS 8-1 against The Trouble with Tribbles
WINS 9-2 against The Devil in the Dark​

Balance of Terror:

WINS 4-2 against Journey to Babel
WINS 5-3 against Mirror, Mirror
WINS 5-3 against Amok Time
WINS 6-1 against The Trouble with Tribbles
WINS 8-2 against The Devil in the Dark​

Journey to Babel:

WINS 5-4 against Mirror, Mirror
WINS 5-3 against Amok Time
WINS 5-3 against The Trouble with Tribbles
WINS 8-1 against The Devil in the Dark​

Mirror, Mirror:

WINS 6-4 against Amok Time
WINS 7-2 against The Trouble with Tribbles
WINS 9-3 against The Devil in the Dark​

Amok Time:

WINS 6-3 against The Trouble with Tribbles
WINS 7-1 against The Devil in the Dark​

The Trouble with Tribbles:

WINS 5-3 against The Devil in the Dark​

 

[Captrek]

Here are the results updated with Space Therapist’s changes.

 

[Isis]

...You know, for shits and giggles, you could just peruse the top ten threads, format ballots yourself from those posts, and then post the results here.

Certainly fascinating, but shouldn't a graph with those results get an asterisk? Those threads ask for a certain number of episodes to be rated. Posters may have strong ideas on other episodes, as indicated by the thread that asked for the top 14 or 15 eps rather than the top 10. However, numbers 11 to 15 in that thread (although I haven't checked yet whether you used that particular one) would not be represented in your calculations/graph. Granted, I find the latest results intriguing, but aren't they not quite in line with the ranking system you originally presented to us?

 

[Isis]

^ ^ ^ ^

...I’ve made ballots for the posts in [THREAD=147974]these[/THREAD] [THREAD=148915]two[/THREAD] threads. I replaced the “truncate” feature (which nobody had used) with an “all others” feature to accommodate the “sandwich” thread...

Yes, captrek, the link with the word "two" accesses the thread I was talking about. (Note thread title should refer to top 14, not top 10.)

 

[Captrek]

Certainly fascinating, but shouldn't a graph with those results get an asterisk?

Not really. The Schulze method is pretty neat. It takes whatever information is available about voter preferences and smartly constructs a ranking (or Hasse diagram in some cases, such as the instant one) based on that information.

The one thing that might warrant an asterisk is the claim that these results reflect the opinions of 16 voters. Put an asterisk on that 16, because 11 of those 16 have provided very limited information about their preferences. The tangled areas near the middle of the graph are based almost entirely on the preferences of just 5 voters (you, me, jk82, CarbonCopy, and Space Therapist).

I find the latest results intriguing, but aren't they not quite in line with the ranking system you originally presented to us?

I realize now that I didn’t express the system as clearly as I thought I had. What I intended to express is that anyone can participate by providing as much or as little information about her preferences as she desires. She can rate all the episodes, or some of them, or just her favorites. She can rate each episode precisely on a scale of 0-100, or create a complete ranking of all episodes, or simply categorize them broadly as good, fair, and poor. The more information a voter expresses about her preferences, the more she can influence the results, but if she wants to make only a small contribution to the results, that’s okay.

ETA: Come to think of it, the sandwich thread is very similar to what Alice does in Example #1, and the Top 10 thread is very similar to what David does in Example #4, in the instructions of the ranking system I originally presented to you.

 

[Isis]

Thanks for the additional explanation, captrek. I think there's one point I'm still not clear on, so if you would bear with me for a little while longer, here goes...

...What I intended to express is that anyone can participate by providing as much or as little information about her preferences as she desires...

OK, say I can rank my top 7 episodes from 1 to 7 (no ties for the purpose of this discussion) and that I also consider them my all-time favorite TOS episodes, way above all the others. I come upon a top 5 thread and list numbers 1 to 5. You then incorporate those results in this thread. However, if I had responded directly to your thread, I would have included eps 1 to 7. My question then is, "If only my top 5 are incorporated but I have seven favorites, aren't the thread results in this thread slightly 'off' because they don't take my numbers 6 and 7 into account?" Does that make any sense? Am I missing something?

 

[Captrek]

Thanks for the additional explanation, captrek. I think there's one point I'm still not clear on, so if you would bear with me for a little while longer, here goes...

...What I intended to express is that anyone can participate by providing as much or as little information about her preferences as she desires...

OK, say I can rank my top 7 episodes from 1 to 7 (no ties for the purpose of this discussion) and that I also consider them my all-time favorite TOS episodes, way above all the others. I come upon a top 5 thread and list numbers 1 to 5. You then incorporate those results in this thread. However, if I had responded directly to your thread, I would have included eps 1 to 7. My question then is, "If only my top 5 are incorporated but I have seven favorites, aren't the thread results in this thread slightly 'off' because they don't take my numbers 6 and 7 into account?" Does that make any sense?

Yes, it makes sense, assuming you define “off” as “something other than the results that would be produced if all expressed preferences taken into account were expressed in direct response to this thread.”

If you’re thinking of a less question-begging concept of offness, I think I need you to articulate it better.

 

[Isis]

Thanks for your response, captrek. What I'm asking is if you take results from a thread that arbitrarily limits the number of episodes a person can rank (eg, sets a "limit" of 10 episodes), are you entering incomplete information into your ranking system? In this same vein, my definition of "off" would be "deficient in some way, such as not being complete." FYI, it really doesn't matter whether or not the preferences, as you stated things, "were expressed in direct response to this thread."

Admittedly, what I'm saying at this point may be clear as mud. If that's the case, I withdraw my questions. Your ranking system was meant to be interesting and fun. So if I have one small question that can't be answered via the message board format, I'm happy to forget about it.

captrek, thanks for bringing up this way to rank episodes in the first place. Maybe others will provide some additional rankings.

 

[Captrek]

Thanks for your response, captrek. What I'm asking is if you take results from a thread that arbitrarily limits the number of episodes a person can rank (eg, sets a "limit" of 10 episodes), are you entering incomplete information into your ranking system? In this same vein, my definition of "off" would be "deficient in some way, such as not being complete."

OK, that makes some sense. Using arbitrarily limited information about the preferences of some posters is “deficient in some way,” but I think it’s less deficient than using no information about the preferences of those same posters. I turned to those other threads out of frustration because we had had only four (now five) ballots posted in this thread. It’s not an ideal solution, but it’s better than nothing. The 16 deserves an asterisk, because we’re working with much less information than we would have if those same 16 posters had all voted “properly.”

Maybe others will provide some additional rankings.

I surely hope so, but I’m not holding my breath.

 

[Isis]

^ ^ ^ ^
Thanks captrek. And I want to be clear I wasn't criticizing the method; I just wanted to learn more about it.

...I turned to those other threads out of frustration because we had had only four (now five) ballots posted in this thread...

I absolutely feel your frustration. At this point, two sayings come to mind: "There's no accounting for taste" and "It is what it is."

And nevertheless, the results that were provided were interesting (both with and without the other threads ). captrek, thanks for posting the method in the first place.

 

[Captrek]

I added 34 ballots from this old [THREAD=140823]top 10 thread[/THREAD].

Note that, being a top-10 thread, it gives an advantage to love-it-or-hate-it episodes (which appear on some top-10 lists) over episodes that are generally considered average (and therefore appear on few if any top-10 lists). Is this the kind of thing you meant by “off”?

Note also that the lower half of the graph remains tangled.