College Policy Debate Forums
January 23, 2019, 04:27:43 PM *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
   Home   Help Search Login Register  
Pages: [1]
Author Topic: Ordinal MPJ and judge diversity -- some facts and some thoughts.  (Read 3547 times)
Posts: 38

« on: August 31, 2016, 10:22:42 PM »

Below is also in the attachment for those who might prefer to print and read later.

I’m writing this just because I’ve had this conversation, using this data, with several people individually over the past few years and I wanted to share what I know with the community.  We, collectively, seem not to yet have found a clear path to address diversity, especially diversity in the judge pool, and a very open question is what role mutually-preferred judging (MPJ) ought to play.
Please note that these opinions are my own, and in all instances I have been deferring to the tournament directors to set tournament policy on diversity and how to, for example, comply with the CEDA rule.
These are what I take to be the facts:
1)   About 70% of the judge pool is composed of white males.  If you would like to check, download the judge lists of any major tournament and mark anyone you know not to be a white male and divide by the total number of judges.  I have done this many times, with many people, and the number of diverse judges always comes up as roughly 30% (the term “diverse” here used in reference to the CEDA rule, which focuses on ethnicity and biological sex).  By way of comparison, the 2016 Republican National Convention speaker list had 40% female speakers.

2)   Diversity opt-in gets about 5% of the judge pool into the category.  At GSU in 2015, 9 of 138 judges opted-in.  At Texas in 2016 it was 8 of 149.  These are entirely typical.   This means that about 5 in 6 judges who might opt-in do not, for whatever reason.

3)   Diverse judges are more likely to be at the bottom of the MPJ pile.  At Texas last year, 5 of the top 25 judges had some non-white male heterosexual characteristic (2 females, 2 African-Americans, 1 openly queer male; obviously, there may be some judges with queer identities that I am unaware of.)  The bottom-25 judges had 11 such people (3 african-american males, 3 african-american females, and 5 female judges with another ethnicity).  Practically speaking, the bottom-25 judges (of any demographic variety) are the most difficult to place for any tab room, and if you are going to lose rounds of judging it is probably from that group.  Note that this largely defeats a system of adding a “bonus” to the prefs of diverse judges (improving the ratings of diverse judges by, say, 10%).  Seven of the 8 opt-in judges fell between the top-25 and bottom-25 (1 opt-in judge was in the bottom 25). 

This is the conclusion that I believe logically flows from these facts:  Opt-in, by itself, will not make any meaningful change to judge assignments.  Participation rates are so low that it will only really change about 2-3 judge placements per tournament, and even if every opt-in judge is placed in elims the maximum possible amount of diversity is one diverse judge on half of the doubles panels.  Practically, there will be more than that because several diverse judges who did not opt-in will be placed, but that’s the most an affirmative system of judge placement can do if it is based on opt-in.
This means that any tab room will face a number of uncomfortable choices:
1)   Do you take affirmative steps to place diverse judges even if they don’t opt-in?  (See the note on the CEDA rule below; the language gives the tab room wide latitude in how it will place diverse judges and it doesn’t require that the judges themselves opt-in.) 
a.   If you use opt-in it will make little practical difference.  If you use your own judgment about who is diverse you can do much more to include diverse judges, but you are undoubtedly on dangerous ground and might ascribe an identity to someone that is different from how they see themselves. 
b.   One good reason that some eligible judges don’t opt-in is that they are overworked and exhausted; many non-diverse judges don’t want to judge for the same reason.  Insisting that diverse judges put in extra effort that they don’t want to because they are diverse seems counter-productive (“we are improving diversity by making diverse judges work harder than everyone else”).  This makes tab-identifying diverse judges even more fraught with difficulty.
c.   Some judges who have diverse demographic characteristics have very traditional argument preferences.  I’m not saying they shouldn’t, I’m just saying that if you are tabbing you feel weird about achieving diversity with the placement of people who are demographically but not argumentatively diverse and who did not opt-in as diverse judges.  It becomes very difficult to figure out the ultimate goal of the CEDA rule: If the value of demographic diversity is that non-white-heterosexual-male judges bring a different perspective on things, the value is not being advanced by placing diverse judges who are preferred because they have a traditional judging perspective.  Frankly, if there were a large number of diverse judges, this wouldn’t be an issue, but there aren’t, and right now diverse judges are much more likely to be a highly preferred and place-able if they have more traditional argument preferences.
2)   Should you spread diverse judges across elim panels or concentrate them in a few?  Often, individual elim panels will have lots of diverse representation because of the judge preferences of both teams.  Simply letting the prefs set the panels will therefore get a large number of diverse judges on a very few panels, creating a large number of all-white-heterosexual-male panels in other debates.  This has the upside of not “diluting” the perspective of the diverse judges, but also has the effect of segregating that perspective into a very small number of debates (and probably those debates where a diverse perspective is already the majority opinion).
3)   Does judging “good” rounds matter?  You can probably place all the diverse judges somewhere, especially out-of-it open debates or JV/novice debates (depending on the operating ADA rule at the time).  Are you just trying to place diverse judges in some round, or is it important that they hear break rounds/highly-seeded prelims/elim rounds?  Saying the quality of the round does count means you are going to have to reach into the bottom-25 judges and put them in a round where the teams’ preferences are low and probably below established thresholds.  To place judges but not be concerned with the quality of the round certainly seems to invoke a glass ceiling: We respect you and welcome your diverse perspective, just not in any round that matters a whole lot.
What are we all to do?
Recruitment: Yes, the fundamental problem is that 70% of the judge pool is not diverse.  Nothing a tab room can do by itself will likely change that (although I do believe tab decisions play a role in making things better).  Here’s my comment: Since my teams started running critical arguments in 2002 our opponents have been swearing that they will leave no stone unturned to recruit diverse students.  Here we are a decade and a half later, 4 full generations of students have come and gone, and any progress is negligible and hardly seems like it’s due to the tireless recruiting efforts of the less diverse programs.  The UDLs have graduated literally tens of thousands of diverse students interested in debate since then.  If we were collectively serious about solving this problem with recruitment it would have been fixed a decade ago.  My twin beliefs on this matter:  Recruitment of diverse people MUST be a central fix to the problem.  Recruitment, by itself, is NOT the only issue at play here.
Tabbing: We either need to do something to incentivize judges to opt-in (One less round of commitment?  Paid a stipend? Simply encouraged folks take an active role in improving diversity?) or clarify when tab rooms might “affirmatively place” someone who has not opted-in.
MPJ:  Look, I like it.  I have worked very hard to come up with systems that maximize it.  But I’ll say this: When the Council of Tournament Directors formed after the 2009 Wake Forest developmental conference, we had people like Ross Smith in the room and very little difficulty agreeing that MPJ was only one goal for judge placement and that other priorities – such as judge development – should also count a lot.  From what I can tell, there is now a much stronger feeling that teams should get their highest pref possible and that it’s better just to have the computer do its thing and leave it at that.  Competitive success in debate thrives on being part of the big rounds, either as a competitor or judge, and getting diverse people into those debates is critical to advance the core goals of diversity in the first place.  We need MPJ systems that BOTH put in the judges that both teams prefer AND rotate judges around so that more judges get exposed to more teams and more vice versa.  A problem we talked about in 2009 – but have not addressed in any meaningful way – is how to address judge “compression,” that is, a very small group of judges hearing almost all the big rounds.  Finally, I want to say that I have composed strike cards at the NDT that had 6 of 11 judges fall into the diverse category, and got the strike cards back with all 6 diverse judges struck.  MPJ plays a role in exclusion.
My answers about tabbing, tentative as they are at this point:
As I mentioned, these views are my own, and in all cases I will defer to the director’s decisions.  But I offer these as food for thought.
Based on my practical experience, the issue for prelims is quite a bit different than elims.  More or less, diverse judges hear about the same percentage of their commitment and the same number of break rounds or rounds involving teams that clear as non-diverse judges.  You gotta watch it, but the tweaks are small and everything is within normal MPJ range.  This is not to discount the value of prelims, but from a tabbing perspective the real difference in the placement of diverse judges becomes much more pressing in the elims.
The real issue, therefore, is elim panels, and that mostly only means doubles and octos.  After that point, the number of judges is so small that percentages start to lose meaning and the main constraint is that most judges are leaving.  So the key question is how you are going to place judges in doubles and octos.
The no-brainer is that you should review all the opted-in judges and make sure that if they fit normal MPJ criteria they get placed.  Another no-brainer is that when you are making the fine judgment calls and you are choosing between placing a diverse and a non-diverse judge, you should place the diverse judge.  That particular choice is more rare than you’d think.  These are worth doing, but by themselves are not going to have much of an impact on the overall diversity of the judge placements.
Using 5-judge panels means that more diverse judges, and more judges overall, get placed.  I view this as a core community good.  The more judges who hear elim debates, especially at the majors, the more people are included at the point that is most meaningful.  I am 100% convinced that this makes people better judges, and close to 100% convinced that it makes them feel more included.  I know for sure that when I was a young (and very bad judge) hearing elim debates really helped me figure out the game more, kept me connected to the whole activity, and made me a much better coach.   Our activity writ large is stronger when we have more programs with more judges hearing elim rounds.  Watching the video is not the same.  If you’ve got enough judges to do 5-judge panels through octos, I think you should.
If there is one truth we have learned over the last 150 years of competitive debate, it is that the activity can survive even if debaters think that their judges voted the wrong way, were too close-minded, were too stupid, or failed to vote for their own brilliant arguments due to a fundamental cognitive flaw in the judge’s basic constitution.  But our activity cannot thrive without a large number of committed and engaged judges and coaches, and we should spend far more time worry about developing the judges into coaches than we should worrying that some rounds that have been lost might have been won with a different set of judges.
I think it is worth looking below the 50th percentile to include diverse judges on elim panels.  Ideally, if you’ve got a judge who’s in the 50-60 range but highly mutual, I think that’s a fine placement.  Practically, the issue is rarely that if you go down to the 55th percentile you’ll find a highly mutual judge…it’s much more likely that you need to go down to about the 75th percentile to find a diverse judge that is mutual.  I am genuinely divided on what to do here, although I will say that if it’s the 5th judge on a 5-judge panel it is much easier to do something to include them than if it is the 3rd judge on a 3-judge panel.
I think it is worth doing what is reasonable to avoid all-white-male elim panels, even if this means that tabroom placing judges who have not identified themselves as diverse.  First, judges serve as role models, and it is easier for diverse debaters to believe they can succeed if they see other people like them in positions of importance.  Second, the value of diversity is that people with different characteristics bring different intellectual perspectives to the table.  Even placing, say, a female judge with traditional argument preferences on a panel makes a difference to the judge, the competitors, and the audience. 
Thank you very much if you’ve read this far.  The take home points are (a) we seriously need to recruit more diverse people into our activity (b) I think 5-judge elim panels are a community good (c) I think tab rooms should do what they can to avoid all-white-male elim panels, and (d) the current CEDA rule and the system of opt-in are unlikely to succeed without a serious re-evaluation of what we’re doing and what we’re trying to accomplish.
Appendix: about the CEDA rule: 
You can read it for yourself here --
The most relevant phrase to this discussion is “For a tournament to receive CEDA sanctioning and qualify for CEDA points, the tournament Director must identify in the invitation if a judge preference system is to be used in assigning judges. The preference system must include a method to affirmatively place racial minorities in judging assignments.”  The phrase is later repeated for affirmatively placing women.
I will assert that this rule is generally not followed.  In my random scan of the invites I could find most included no diversity statement at all or simply referenced the CEDA rule saying that they would follow it, although my reading is that following the rule means specifying in the invite what you intend to do.  Since January of 2014, 29 tournaments have checked the box in tabroom to track judge diversity; there have been 189 tournaments in that time.
I will also say that the rule says nothing about opt-in, and leaves room for any tab room to affirmatively place diverse judges whether they opt-in or not.
Appendix: notes: In the discussion above I am not reporting data for tab rooms that I have not been a part of other than to note some aggregate settings that are easily checked in the tabroom database.  Judges can change their diversity designations during the course of the tournament, so the final count of opt-in judges is more than a little fluid and can change over the course of a tournament.   I’m including non-heterosexual identities in my discussion above even though they are not in the CEDA rule because those issues are central to national politics and having a homosexual identity decidedly exposes you to many societal exclusions.  I realize my discussion is a little inconsistent and fluid on the things that makes one diverse; I simply hope that we can move forward on how with grapple with diversity questions in the judge pool without a broad-based consensus on a unified theory of identity.

* judge diversity.docx (21.08 KB - downloaded 207 times.)
Jr. Member
Posts: 75

« Reply #1 on: September 01, 2016, 12:45:47 PM »

Great read, Jon. Thanks for continuing to contribute to this discussion. I am going to take this opportunity to resubmit my far-too-long explanation for why I think most of the methods that we have used so far incentivize behavior to the detriment of diversity goals and offer a different idea.

Before I get to my post from 5 years ago, I just wanted to say one thing about 5 judge panels. If you combine the 5 judge panels with your numbers about existing diversity, the result is almost always to dilute the impact/input of the judge(s) that are diverse. Perhaps we could tier the panels such that if increasing past 3 does not decrease diversity for a panel, then it can increase to 5. Of course, the problems with identifying and describing diversity are still present, but that is all in the rest of the details.

Also, if you focus on elims, it sort of begs the question about the makeup of the teams that are in elims to receive preferences. I think there is a better way.

Since the last time I posted this, I think only about 3 people read it, I will just give a link to the full document. It is about . I will try to give a more brief description of the proposal here:

Do judge preference affirmative action at the team level. Teams that include one or more people that identify in categories of which we are trying to increase recruitment/retention get a bump in the weight of their preference sheet in the calculation of MPJ. The primary reason this works better is because it is much more difficult to game, and removes the adverse selection incentive to place any judge that might be considered diverse but also that is divergent from your own argument preferences at the absolute bottom of the pool. This is because the other judges that are argumentatively divergent from your preferences, but also not in a potential protected category would now be more likely to be your judge.

I think that the idea that we treat preferences as neutral is patently absurd. The idea that a given percentile of judge is similar in preference to another team's judge at that percentile is obviously false. When factoring race/gender into the reasons that there is a lack of similarity, it should be obvious that we should not treat the percentiles as the same.

One final thing that I will add here is that the supply problem magnifies the need to do this at the team level. Often there are not enough judges to improve preferences for all the teams that probably need a corrective. Only doing preferences at a team level fixes this problem.

Anyway, if anyone is interested. The original post and some discussion is at:,6066.msg13472.html

Posts: 38

« Reply #2 on: September 01, 2016, 01:02:56 PM »

Really enjoyed reading your thoughts...and this is a very creative approach I hadn't thought about.  I'd be happy to code the option in if I can get the go-ahead from some tournament or organization that supports it.

One thought about gaming and the identity question: The way to game this would be to encourage judges with our own ideological bent to "opt in" as diverse...if there are, say, 8 racially diverse judges you don't like, and you can get 8 judges you do like to opt-in as diverse on some other identity criteria (or even as a broadly self-identified ethnic identity), doesn't this end up as a wash?

Your description of the various goals of diversity I found especially useful when thinking about this...
« Last Edit: September 01, 2016, 01:22:38 PM by jbruschke » Logged
Jr. Member
Posts: 75

« Reply #3 on: September 01, 2016, 03:59:57 PM »

I will inquire about tournaments/orgs. I am sure UMKC in 2 weeks is too soon...

You are right that you could reverse-game the judge opt-in, but the organization of it would be so cumbersome, it would surely get exposed. It would be limited to the teams that knew about it, and what if they disagreed about the quality of those otherwise-majority-category judges...

The elegance and simplicity of the team-centered approach is what I like about it--even if the coding is not itself simple or elegant.

Posts: 38

« Reply #4 on: September 02, 2016, 04:35:21 PM »

It's certainly an idea worth pursuing.

The worry is that the small number of opt-in judges makes the counter-balance rather possible; if you know there are only 8 opt-in judges you just need to find 8 females/non-Caucasians in your top 30 judges and encourage them to opt-in...and I don't even think that's nefarious.  It is an attempt to show that people of different biological sexes and ethnicities like the way you debate. You just have to drop them a line and say "Hey, we're preffing you anyway, would you mind opting in so we're more likely to get you?" 

Another thing to consider is whether you would need to release, at least to the teams, their diversity-adjusted prefs.  If you don't things aren't that transparent and the judge you thought you ranked a 60 the computer might think of as 45.

Finally, the current system is that judges can opt in and out at any point...they do the setting on their own account and tabroom just reads it.  If the prefs are going to be adjusted the diversity status probably needs to be fixed at some point, and that might involve more than a small amount of recoding.

I offer all these as wrinkles that need to be ironed, not as deal-breakers.
Full Member
Posts: 165

« Reply #5 on: September 12, 2016, 11:12:36 AM »

Enjoyed reading this back and forth - two thoughts

1) The CEDA Rule: CEDA has had a hard time interpreting and enforcing the rule. In my opinion the amendment was definitely submitted a couple years too early and may have actually created a stop-gap in the conversation about diverse judging, as opposed to advancing it. This is not a criticism of whoever submitted the amendment, I believe they had the best intent. CEDA has ultimately interpreted the rule in the most generous way possible, since the guidelines presented in the amendment are quite vague its hard to deny tournaments sanctioning unless they truly do nothing. Long term I imagine CEDA deciding on a finite number of diversity enhancing options and amending the amendment to provide clarity.

2) Vega's Plan - I'm a big fan. I could say a lot about it but I think Vega is, in a word, thorough in his defense.

Thanks so much for posting,
Pages: [1]
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!