The ultimate episode ranking thread

[Captrek]

Here are the results updated with Space Therapist’s changes.

[Isis]

CarbonCopy wrote:

---

...You know, for shits and giggles, you could just peruse the top ten threads, format ballots yourself from those posts, and then post the results here.

Certainly fascinating, but shouldn't a graph with those results get an asterisk? Those threads ask for a certain number of episodes to be rated. Posters may have strong ideas on other episodes, as indicated by the thread that asked for the top 14 or 15 eps rather than the top 10. However, numbers 11 to 15 in that thread (although I haven't checked yet whether you used that particular one) would not be represented in your calculations/graph. Granted, I find the latest results intriguing, but aren't they not quite in line with the ranking system you originally presented to us?

[Isis]

^ ^ ^ ^

captrek wrote:

---

...I’ve made ballots for the posts in these two threads. I replaced the “truncate” feature (which nobody had used) with an “all others” feature to accommodate the “sandwich” thread...

Yes, captrek, the link with the word "two" accesses the thread I was talking about. (Note thread title should refer to top 14, not top 10.)

[Captrek]

Isis' Evil Twin wrote:

---

Certainly fascinating, but shouldn't a graph with those results get an asterisk?

Not really. The Schulze method is pretty neat. It takes whatever information is available about voter preferences and smartly constructs a ranking (or Hasse diagram in some cases, such as the instant one) based on that information.

The one thing that might warrant an asterisk is the claim that these results reflect the opinions of 16 voters. Put an asterisk on that 16, because 11 of those 16 have provided very limited information about their preferences. The tangled areas near the middle of the graph are based almost entirely on the preferences of just 5 voters (you, me, jk82, CarbonCopy, and Space Therapist).

I find the latest results intriguing, but aren't they not quite in line with the ranking system you originally presented to us?

I realize now that I didn’t express the system as clearly as I thought I had. What I intended to express is that anyone can participate by providing as much or as little information about her preferences as she desires. She can rate all the episodes, or some of them, or just her favorites. She can rate each episode precisely on a scale of 0-100, or create a complete ranking of all episodes, or simply categorize them broadly as good, fair, and poor. The more information a voter expresses about her preferences, the more she can influence the results, but if she wants to make only a small contribution to the results, that’s okay.

ETA: Come to think of it, the sandwich thread is very similar to what Alice does in Example #1, and the Top 10 thread is very similar to what David does in Example #4, in the instructions of the ranking system I originally presented to you.

[Isis]

Thanks for the additional explanation, captrek. I think there's one point I'm still not clear on, so if you would bear with me for a little while longer, here goes...

captrek wrote:

---

...What I intended to express is that anyone can participate by providing as much or as little information about her preferences as she desires...

OK, say I can rank my top 7 episodes from 1 to 7 (no ties for the purpose of this discussion) and that I also consider them my all-time favorite TOS episodes, way above all the others. I come upon a top 5 thread and list numbers 1 to 5. You then incorporate those results in this thread. However, if I had responded directly to your thread, I would have included eps 1 to 7. My question then is, "If only my top 5 are incorporated but I have seven favorites, aren't the thread results in this thread slightly 'off' because they don't take my numbers 6 and 7 into account?" Does that make any sense? Am I missing something?

[Captrek]

Isis' Evil Twin wrote:

---

Thanks for the additional explanation, captrek. I think there's one point I'm still not clear on, so if you would bear with me for a little while longer, here goes...

OK, say I can rank my top 7 episodes from 1 to 7 (no ties for the purpose of this discussion) and that I also consider them my all-time favorite TOS episodes, way above all the others. I come upon a top 5 thread and list numbers 1 to 5. You then incorporate those results in this thread. However, if I had responded directly to your thread, I would have included eps 1 to 7. My question then is, "If only my top 5 are incorporated but I have seven favorites, aren't the thread results in this thread slightly 'off' because they don't take my numbers 6 and 7 into account?" Does that make any sense?

Yes, it makes sense, assuming you define “off” as “something other than the results that would be produced if all expressed preferences taken into account were expressed in direct response to this thread.”

If you’re thinking of a less question-begging concept of offness, I think I need you to articulate it better.

[Isis]

Thanks for your response, captrek. What I'm asking is if you take results from a thread that arbitrarily limits the number of episodes a person can rank (eg, sets a "limit" of 10 episodes), are you entering incomplete information into your ranking system? In this same vein, my definition of "off" would be "deficient in some way, such as not being complete." FYI, it really doesn't matter whether or not the preferences, as you stated things, "were expressed in direct response to this thread."

Admittedly, what I'm saying at this point may be clear as mud. If that's the case, I withdraw my questions. Your ranking system was meant to be interesting and fun. So if I have one small question that can't be answered via the message board format, I'm happy to forget about it.

captrek, thanks for bringing up this way to rank episodes in the first place. Maybe others will provide some additional rankings.

[Captrek]

Isis' Evil Twin wrote:

---

Thanks for your response, captrek. What I'm asking is if you take results from a thread that arbitrarily limits the number of episodes a person can rank (eg, sets a "limit" of 10 episodes), are you entering incomplete information into your ranking system? In this same vein, my definition of "off" would be "deficient in some way, such as not being complete."

OK, that makes some sense. Using arbitrarily limited information about the preferences of some posters is “deficient in some way,” but I think it’s less deficient than using no information about the preferences of those same posters. I turned to those other threads out of frustration because we had had only four (now five) ballots posted in this thread. It’s not an ideal solution, but it’s better than nothing. The 16 deserves an asterisk, because we’re working with much less information than we would have if those same 16 posters had all voted “properly.”

Maybe others will provide some additional rankings.

I surely hope so, but I’m not holding my breath.

[Isis]

^ ^ ^ ^
Thanks captrek. And I want to be clear I wasn't criticizing the method; I just wanted to learn more about it.

captrek wrote:

---

...I turned to those other threads out of frustration because we had had only four (now five) ballots posted in this thread...

I absolutely feel your frustration. At this point, two sayings come to mind: "There's no accounting for taste" and "It is what it is."

And nevertheless, the results that were provided were interesting (both with and without the other threads ). captrek, thanks for posting the method in the first place.

[Captrek]

I added 34 ballots from this old top 10 thread.

Note that, being a top-10 thread, it gives an advantage to love-it-or-hate-it episodes (which appear on some top-10 lists) over episodes that are generally considered average (and therefore appear on few if any top-10 lists). Is this the kind of thing you meant by “off”?

Note also that the lower half of the graph remains tangled.

[Isis]

captrek wrote:

---

...Note that, being a top-10 thread, it gives an advantage to love-it-or-hate-it episodes (which appear on some top-10 lists) over episodes that are generally considered average (and therefore appear on few if any top-10 lists). Is this the kind of thing you meant by “off”?...

OK, captrek, here's another way to express my point, which I would say at this point is a minor one. Of episodes that do not appear in a rater's ranking, the more the rater likes the episode, the shorter shrift it gets in the poll. To give an example, say I list my top ten episodes in a "Top Ten" thread. Say "Errand of Mercy" is my #11 and "And the Children Shall Lead" is my #79, but because these are not "Top Ten" episodes for me, they don't appear in the ranking I provided. Say you incorporate those results in this thread. I'm saying the "Errand of Mercy" gets short shrift. It's my #11, but in your poll, it's handled the same way as my #79. Does this shed any light on things?

captrek wrote:

---

...Note also that the lower half of the graph remains tangled.

Although this can be taken with a "grain of salt" because we're still not talking about tons of people, there seem to be some consistencies in people's preferences. In particular, the first seven in your latest diagram have always come out pretty high. (This isn't meant to be a scientific statement, just an overall general "take" on things.)

[CorporalCaptain]

It makes sense to me, Isis.

But the conclusion one gets from the point is that in order to get an accurate ballot, the voter contributing it must answer 79 question.

If you don't provide a number for every episode, the ballot will suffer from this problem.

[Captrek]

Isis wrote:

---

OK, captrek, here's another way to express my point, which I would say at this point is a minor one. Of episodes that do not appear in a rater's ranking, the more the rater likes the episode, the shorter shrift it gets in the poll. To give an example, say I list my top ten episodes in a "Top Ten" thread. Say "Errand of Mercy" is my #11 and "And the Children Shall Lead" is my #79, but because these are not "Top Ten" episodes for me, they don't appear in the ranking I provided. Say you incorporate those results in this thread. I'm saying the "Errand of Mercy" gets short shrift. It's my #11, but in your poll, it's handled the same way as my #79. Does this shed any light on things?

Yes and no.

The poll summarizes expressed preferences, not actual preferences. (Unfortunately, we don’t have Kor’s mind sifter.) If your ballot only lists your top 10, then your favorites of the remainder get “short shrift” in the same sense that your favorites get short shrift if you don’t cast a ballot at all.

The Schulze Method handles this incomplete information pretty well. Your ballot in this example doesn’t say that EOM and ATCSL are “the same,” and doesn’t push EOM down and ATCSL up toward mediocre. Your ballot simply expresses no opinion at all about EOM vs. ATCSL.

This “ultimate” ranking has failed to live up to that title. It is based on very incomplete information. Dozens of posters have contributed no information about their preferences beyond their top 10, and millions of Star Trek fans have contributed no information at all about their preferences. The graph is a synthesis of the information that is available on these ballots, and shouldn’t be represented or read as anything more than that.

[Isis]

CorporalCaptain wrote:

---

It makes sense to me, Isis.

But the conclusion one gets from the point is that in order to get an accurate ballot, the voter contributing it must answer 79 question.

If you don't provide a number for every episode, the ballot will suffer from this problem.

Corporal, I would just make one change to what you said; I'd change the beginning of your second sentence to read, "But the conclusion one gets from the point is that in order to get the most accurate ballot..." Otherwise, I absolutely agree with you.

[CorporalCaptain]

Quite.