Via The Volokh Conspiracy, I enjoyed this discussion of the NFL instant replay system. A call made on the field can only be overturned if the replay reveals conclusive evidence that the call was in error. Legal scholarship has debated the merits of such a system of appeals relative to the alternative of de novo review: the appelate body considers the case anew and is not bound by the decision below.
If standards of review are essentially a way of allocating decisionmaking authority between trial and appellate courts based on their relative strengths, then it probably makes sense that the former get primary control over factfinding and trial management (i.e., their decisions on those matters are subject only to clear error or abuse of discretion review), while the latter get a fresh crack at purely “legal” issues (i.e., such issues are reviewed de novo). Heightened standards of review apply in areas where trial courts are in the best place to make correct decisions.
These arguments don’t seem to apply to instant replay review. The replay presumably is a better document of the facts than the realtime view of the referee. But not always. Perhaps the argument against in favor of deference to the field judge is that it allows the final verdict to depend on the additional evidence from the replay only when the replay angle is better than that of the referee.
That argument works only if we hold constant the judgment of the referee on the field. The problem is that the deferential system alters his incentives due to the general principle that it is impossible to prove a negative. For example consider the (reviewable) call of whether a player’s knee was down due to contact from an opposing player. Instant replay can prove that the knee was down but it cannot prove the negative that the knee was not down. (There will be some moments when the view is obscured, we cannot be sure that the angle was right, etc.)
Suppose the referee on the field is not sure and thinks that with 50% probability the knee was down. Consider what happens if he calls the runner down by contact. Because it is impossible to prove the negative, the call will almost surely not be overturned and so with 100% probability the verdict will be that he was down (even though that is true with only 50% probability.)
Consider instead what happens if the referee does not blow the whistle and allows the play to proceed. If the call is challenged and the knee was in fact down, then the replay will very likely reveal that. If not, not. The final verdict will be highly correlated with the truth.
So the deferential system means that a field referee who wants the right decision made will strictly prefer a non-call when he is unsure. More generally this means that his threshold for making a definitive call is higher than what it would be in the absence of replay. This probably could be verified with data.
On the other hand, de novo review means that, conditional on review, the call made on the field has no bearing. This means that the referee will always make his decision under the assumption that his decision will be the one enforced. That would ensure he has exactly the right incentives.
Cheap Talk 
3 comments
Comments feed for this article
December 15, 2009 at 2:09 pm
davidcroushore
This is an interesting subject. From my (ridiculous number of) hours watching football, it often seems as if referees tend towards making calls that are, by rule, unreviewable. The best and most common example is that of the incomplete pass vs. fumble dilemma that occurs almost once per game. In these cases, the referee may call an incomplete pass, which is not reviewable, or a fumble, which is. The call is so difficult that the distinction is often not even clear on replay. It seems (though I don’t know that data exists on this) that referees tend to rule incomplete pass (unreviewable) much more often in these situations. I wonder if there is an incentive basis for this behavior, or if “incomplete pass” is simply a more common call throughout the game, and thus more habitual.
December 15, 2009 at 3:03 pm
Sean Crockett
In Jeff’s example, a call of down-by-contact is not reviewable. So it is indeed with 100% certainty that the call will stand, and the ref will set a higher threshold probability for making a definitive call if he prefers correct calls to be made.
But it is not clear to me that his incentives aren’t “right,” and that under the alternative de Novo review he has “exactly the right incentives”. Yes, he will more heavily weight what he believes to be true under de Novo review. But incorrect definitive calls cannot be corrected by replay while incorrect non-calls can be; thus type 1 errors may impact the outcome of the game while type 2 errors will not (because they are overturned).
So if we want refs to minimize the total number of errors they make, then de Novo review is the better mechanism. If we instead want refs to minimize the number of errors that may change the outcome of the game, the present system is the best mechanism, and we actually want refs to set their threshold probabilities to one. Of course, this would dramatically slow down the pace of the game (as most plays would be reviewed). So I think the best thing for football is the present system, where type 1 errors are induced to be more costly than type 2 errors, but not infinitely so. Maybe induce a threshold probability of 60-65%. Personally, I think this mechanism induces “the right” incentives in referees.
December 22, 2009 at 11:45 am
Trophy Cups
You’re right to bring up the fumble dilemma David-could it be that this is almost a purposeful action on the part of certain individuals?