Sign In

Communications of the ACM

Viewpoint

Effectiveness of Anonymization in Double-Blind Review


View as: Print Mobile App ACM Digital Library In the Digital Edition Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
Effectiveness of Anonymization in Double-Blind Review, illustration

Credit: Lucky Team Studio

Peer review is a cornerstone of the academic publication process but can be subject to the flaws of the humans who perform it. Evidence suggests subconscious biases influence one's ability to objectively evaluate work: In a controlled experiment with two disjoint program committees, the ACM International Conference on Web Search and Data Mining (WSDM'17) found that reviewers with author information were 1.76x more likely to recommend acceptance of papers from famous authors, and 1.67x more likely to recommend acceptance of papers from top institutions.6 A study of three years of the Evolution of Languages conference (2012, 2014, and 2016) found that, when reviewers knew author identities, review scores for papers with male-first authors were 19% higher, and for papers with female-first authors 4% lower.4 In a medical discipline, U.S. reviewers were more likely to recommend acceptance of papers from U.S.-based institutions.2

These biases can affect anyone, regardless of the evaluator's race and gender.3 Luckily, double-blind review can mitigate these effects1,2,6 and reduce the perception of bias,5 making it a constructive step toward a review system that objectively evaluates papers based strictly on the quality of the work.


Comments


Moritz Beller

Very interesting! It seems then that manual reviewers are no better at guessing than using a machine learning-based approach based on the number of times authors appear in the references, see https://peerj.com/preprints/1757v1.pdf.


Charles Weir

Interesting. In a recent double-blind review process we came across one issue: multiple entries by the same author(s), involving self-plagiarisation. The double blind process made this very hard to detect; theoretically a CRP system could detect and warn without revealing author identities, but ours at least did not do so.


Acton Bell

The results reported here are interesting. However, since some of the figures cited in the introductory paragraph are being repeated in online discussions, it seems worth pointing out the following:

1) The 1.76x and 1.67x odds multipliers quoted from Tomkins et al. for the effects of author fame and institution reputation are from an early version of that paper. The numbers from the latest version (v6) on arXiv are smaller (1.63 and 1.58, respectively). This most recent version was probably not available when this Viewpoint was submitted or initially drafted. Version 5, posted May 2017 has factors of 1.66 and 1.61, respectively. Of course, these are still large effects.

2. The confidence intervals for these odds multipliers from Tomkins et al. are quite large: the bottom end of the range for the "author fame" correlation coefficient corresponds to an odds multiplier of 1.05. In light of this, and the changes from different version of the paper, I think one should be careful about placing too much emphasis on their exact size.

3. I think it is worth noting some negative findings from the cited papers, particularly when they contradict the other mentioned positive findings. For instance, the Viewpoint mentions that in a certain medical field, earlier work found a bias from American reviewers in favor of American authors. On the other hand, Tomkins et al. find no significant bias based on shared nationality of reviewers and authors. Additionally, Tomkins et al. do not find a statistically significant bias based on author gender (though they argue that when placed in context of a larger meta-analysis of other studies, these results may be considered significant).


Displaying all 3 comments

Log in to Read the Full Article

Sign In

Sign in using your ACM Web Account username and password to access premium content if you are an ACM member, Communications subscriber or Digital Library subscriber.

Need Access?

Please select one of the options below for access to premium content and features.

Create a Web Account

If you are already an ACM member, Communications subscriber, or Digital Library subscriber, please set up a web account to access premium content on this site.

Join the ACM

Become a member to take full advantage of ACM's outstanding computing information resources, networking opportunities, and other benefits.
  

Subscribe to Communications of the ACM Magazine

Get full access to 50+ years of CACM content and receive the print version of the magazine monthly.

Purchase the Article

Non-members can purchase this article or a copy of the magazine in which it appears.