College Policy Debate Forums
July 22, 2018, 04:44:32 AM *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: IF YOU EXPERIENCE PROBLEMS WITH THE SITE, INCLUDING LOGGING IN, PLEASE LET ME KNOW IMMEDIATELY.  EMAIL ME DIRECTLY OR USE THE CONTACT US LINK AT THE TOP.
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Judge Preference - Thought Experiments  (Read 4898 times)
William Mosley-Jensen
Jr. Member
**
Posts: 79


« on: March 08, 2012, 04:59:22 PM »

So I have been tabbing some tournaments here and there for a while now and want to get a discussion going about judge preference and the assignment of judges.

The question that I have been considering is: should we strive to maximize preference or mutuality? Let me provide a couple of cases to clarify the question to see what folks think.

Case 1 - Large tournament with 160 judges, ordinal prefs. The tab room can either assign a top 20% judge for each team, your opponents ordinal #1 and your ordinal #31, or the tab room can assign a judge just outside of the top 20%, with each team's ordinal 34. Maximize preference or mutuality here?

Case 2 - Large tournament with 160 judges, ordinal prefs. The tab room can either assign a top 20% judge for each team, your opponents ordinal #1 and your ordinal #31, or the tab room could assign a top 30% judge that is more mutual, each teams' ordinal #48. Would you prefer that the tab room maximize preference (and achieve an average preference of 16 or top 10%) or assign the ordinal 48 (a top 30% judge) and maximize mutuality?

If your answer to the first case is mutuality and the second case is preference, where should the line be drawn?
Logged
jamesherndon3
Newbie
*
Posts: 33


« Reply #1 on: March 08, 2012, 05:14:12 PM »

Great conversation to have, thanks Will M-J.

Case 1 - Large tournament with 160 judges, ordinal prefs. The tab room can either assign a top 20% judge for each team, your opponents ordinal #1 and your ordinal #31, or the tab room can assign a judge just outside of the top 20%, with each team's ordinal 34. Maximize preference or mutuality here?

ANSWER - Mutual 34. 
31 to 1 is too big of a gap.  If the gap were smaller/not as close to the mutuality then I'd say top the top 20%.

Case 2 - Large tournament with 160 judges, ordinal prefs. The tab room can either assign a top 20% judge for each team, your opponents ordinal #1 and your ordinal #31, or the tab room could assign a top 30% judge that is more mutual, each teams' ordinal #48. Would you prefer that the tab room maximize preference (and achieve an average preference of 16 or top 10%) or assign the ordinal 48 (a top 30% judge) and maximize mutuality?

ANSWER - Not sure.  I could be convinced either way.  top 30% is probably fine, so I'd take the #48.  if we were talking 40% OR the gap between 1 and 31 was smaller, I'd go the other way.


I like the discussion, and I do think we prioritize mutuality too much [actually think the overuse of the pref sheet in general is a problem].  If, in both cases, the gap was smaller - say 1 to 21 - my answer changes.

I will advocate what I did in our discussion in the D6 tab discussion:
Tournaments should advertise a cut off spot.  All judges who are ranked ordinally above "X" line will be used regardless of mutuality.  Sort of a "pick your top 20."  It is a way for a tournament to say, "we think you should pick 20 judges from our pool that you are fine with judging you regardless."  Benefits:
a.  creates some judge pool flex.
b.  not all mutuality is created equal.  1 to 20 is closer to me than 40 is to 60 in the ranking of judges.  Mutuality is far more important for judges outside the "x"
c.  people should be able to take 20 or so judges and have them in every round.

If one thinks that 30 judges is a better number then they probably like both of your cases.  If one thinks that there aren't 20 good judges then they probably don't like either case.


Logged
tcram
Full Member
***
Posts: 167


« Reply #2 on: March 08, 2012, 05:21:51 PM »

The finer points of judge placement and constraints created the later the tournament wears on are largely beyond me.  As a coach, I would be comfortable with a designated 'top 20', and I also agree with Herndon's belief that the practical difference in the 40 to 60 range is more significant than in the upper tiers.
Logged
William Mosley-Jensen
Jr. Member
**
Posts: 79


« Reply #3 on: March 08, 2012, 07:24:57 PM »

Given that there is some agreement that a team could select a certain amount of judging, say their top 20 critics, that they are fine with having in any circumstance it seems that the justification for using an ordinal system all the way down may not be entirely clear.

In other words, the advantage that a strictly ordinal system has over other ranking schemes is the ability to achieve fine-grained mutuality throughout the preference range. This seems to be a disadvantage if it prevents the assignment of a judge in a teams top range because there is a more "mutual" option.

I have been thinking about a hybrid preference system where a team would rank a certain amount of judges as "1's" and then begin the ordinal system after those rankings. So, lets say with 160 judges, you rank 25 judges as "1's" then begin ranking the rest of the pool after that. This could provide a ready pool of judges deemed to be qualitatively high on a team's sheet, and also provide a discriminating mutuality for judges below that threshold.

Logged
Whit
Jr. Member
**
Posts: 79


« Reply #4 on: March 08, 2012, 09:16:41 PM »

I could be totally off on this, but it is my understanding that the system allows this now in some way. If you don't rank a judge in the ordinal system, then the default rank for that judge is whatever the other team ranked them. So in a sense, they become a 'super one,' because you've told the tab you will take that person no matter what the other team ranks them as (i.e. mutuality is guaranteed for that judge).

Logged
Adam Symonds
Sr. Member
****
Posts: 353


« Reply #5 on: March 08, 2012, 09:46:03 PM »

A 0 is treated by the system as the identical pref for the other team, so it is in effect a Super 1. Of course, if you simply forget to rank some folks, it could be more like a Super Strike that you get if your opponent loves them.
Logged
jamesherndon3
Newbie
*
Posts: 33


« Reply #6 on: March 09, 2012, 07:41:04 AM »

The problem with leaving a judge as a ZERO is that it moves everyone down a spot, right?  So, if I leave my fave favorites as 0 then everyone else is 5 spots higher.  seems to destroy what the goal of actual mutuality is.

Plus, it isn't universally followed so its just another way to 'game' the system to provide guaranteed mutuality. 

Finally, I thought I remember Bruschke telling me that any judges left unranked got changed to a higher number to avoid teams getting accidently unranked judges.  That could obviously be both made up and subjective for each tab person.
Logged
William Mosley-Jensen
Jr. Member
**
Posts: 79


« Reply #7 on: March 09, 2012, 12:41:49 PM »

I believe that the way that ordinals are currently done, even if you ranked several judges as 1's, then you would still have to fill in the rest of the ordinal system normally. Under the old category system, 1-9, A-D, rankings were to assign a certain number of judges in that category or higher. Theoretically a team could assign twice as many 1's as necessary, skip ranking 2's and begin assigning 3's.

Of course, this whole discussion assumes that judge ranking is more of an art than a science, and that the purpose of preference systems is to give us a method to place judges that are within a "best fit" of a suitable range.

The argument could be made that the fine-grained and discriminating nature of ordinals allows us to exactly target our judges. I.E., my number 32 judge is precisely where they should be, not 31 and not 33. If we assume that is the case, then perhaps privileging mutuality is not so bad, as it recognizes the precision we are technologically capable of.
Logged
jgonzo
Jr. Member
**
Posts: 81


« Reply #8 on: March 21, 2012, 01:18:19 AM »

This is an "ask Gary" sort of question, but the problem with any categorical OR ordinal system is its imprecision in actually expressing preference. First, categorical prefs aren't actually categorical variables. They're . . . well, they just are, since they aren't exactly interval variables, either.

Unless I'm wrong about the way that the Gary algoritm works, our ordinal system, while definitely ordinal is ALSO an interval system, which is one of the problems at which Herndon hints. For instance, it assumes that the difference between my 1 and 2 is identical to the difference between 124 and 125, which, obviously, it isn't. Gary attempted to remedy this system with the system that I-don't-remember-a-name-for-but-will-call "assign every a judge a grade between 0 and 99, duplicates are OK."(think 2006-2007). In theory, this system reflected the best of both worlds: it allowed ranking each judge on a relative scale and didn't force us to lump people into unnatural categories (3,6,9, etc). If memory serves, too many people tried to game it by ranking a small number of judges very highly and the rest of the pool as zero or similarly low, then went absolutely apeshit when the tab gave them a zero.

My take is this - the current ordinal system is incredibly effective at producing the outcome for which it is designed. Anybody that complains about that outcome isn't upset about ordinal prefs, they are disagreeing with the intended outcome. Another way to say that is that any system of preference is only as good as the norms that are intentionally reflected in its design.

One of the strong points of the current system (ordinal) over a hybrid top-20, then-ordinal system is that the top-20 cutoff is itself an entirely subjective exercise. I generally do not like subjective judgments being made in the tab room, precisely because of the fact that they don't happen in the light of day. So transparency would be a good thing. If it's going to be top-20, make it quite public beforehand. Second, I would hope that it would be used consistently throughout the season. I think we could come to live with whatever system was in place so long as it was a system we consistently use. What might be warranted is a discussion of what norms we hope to select in our judge assignment system.

J
Logged
PaulStrait
Newbie
*
Posts: 4


« Reply #9 on: April 11, 2012, 12:27:49 PM »

It would be nice if there were data on this question that could be analyzed.  Whether and to what extent the pref sheet can explain any variance in voting patterns of 1/33-like judges I think makes a big difference.  We might have an intuitive feel here, but it seems largely an empirical question.  These sort of data probably exist in some form already.  Beyond voting patterns, another potentially helpful other outcome measure could be some sort of post-post-round satisfaction survey completed by losing teams (I'm thinking something very brief, perhaps 1 or 2 items).    Wouldn't be too difficult to implement if there were the will for this sort of research (though I doubt that there is).
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.0.19 | SMF © 2013, Simple Machines
SMF customization services by 2by2host.com
Valid XHTML 1.0! Valid CSS!