A New Paper About The Editorial Process

By Ivo Welch.  Here is the abstract:

This paper analyzes referee recommendations in two settings: The first setting is a prestigious finance conference, in which a computer algorithm matched referees to papers based only on shared expertise. The second setting is the standard journal process, with data from eight prominent economics and finance journals (ECMTA, JEEA, JET, QJE, IER, RAND, JF, RFS). Despite referee selection differences, the data suggest similar referee behavior in both settings. First, referees display only modest consensus. Second, referees disagree not only about scales (a referee mean effect), but also about the relative ordering of papers. Third, the bias measured by the average generosity of the referee on other papers is about as important in predicting a referee’s recommendation as the opinion of another referee on the same paper.

In sum, the typical referee report consists roughly of one part signal of some referee- agreeable objective attribute of the paper and two parts (referee-specific) noise. In turn, the noise itself consists roughly of one part referee-mean effect (bias) and two parts unidentified effects or noise.