End Anonymous Refereeing

Discussions between computer science researchers often turn to the publishing system, and they invariably reveal widespread dissatisfaction with it. They tend to be longer on criticism than solutions.

One of the first step towards such a solution is to do away with the notion that referees should be anonymous. Who came up with this bizarre idea?

The official justification seems to be that anonymity liberates the reviewer from the threat of career retaliation from a rejected author. It does not stand two minutes’ reflection.

First, under what scenario does a junior person review a senior contributor’s submission? When Marie Curie submits a paper, Max Planck should review it, not a first-year PhD student. It is supposed to be peer review. Those judging you are your equals or betters, who should have nothing to fear from you for their next application for promotion. If anything, you have something to fear from them. And your name is there on the paper.

Worse, this conventional-wisdom justification attempts to deflect a minor risk (protecting the reviewers from an occasional disgruntled author) by creating a much bigger one: the risk of giving cover to nasty reviewers. More likely than  the vengeful scorned author is the case of the devious reviewer, who hides behind anonymity to bad-mouth  a competitor.

In today’s academic world, publication is not just about fostering research creativity to advance human knowledge, but also, increasingly, about something more mundane: advancing one’s career by raising publication and citation counts. Add to this context the peculiar computer science culture, with:

• Its nastiness problem.
• The prime role  of conferences, whose prestige flows largely from how selective they are.

You get a  system designed not to encourage the publication of potentially interesting work but to reject the most papers possible. (Thereby creating a vicious circle: the natural strategy for researchers is to submit a paper to the top-rated conference first, and after rejection try again downwards until finding a venue which deems it good enough. As a result, papers cycle a few times through the refereeing machine, overwhelming the reviewer community and putting ever more pressure to reject. The process also defeats the supposed advantage of conferences, speed of publication, since papers typically undergo several review cycles, each restarted from scratch and delaying publication by four to eight months, prior to seeing the light of day.)

In a more healthy environment, program and editorial committees would try to spot the most interesting papers and help their authors remove imperfections. In the actual environment of CS research, while some of this process (thankfully) happens, the natural incentive is not to find papers to accept but to find the most possible papers to reject. Tight reviewing deadlines do not leave room for second chances. You did not mention possible threat to validity X, I did not understand your section Y (which I phrase in my review as “section Y is poorly written“), you did not cite the work of Z (who happens to be me, but never mind, I am anonymous) — off you go!

Il meschino calunniato,
Avvilito, calpestato,
Sotto il pubblico flagello,
Per gran sorte va a crepar.

(The poor slandered wretch, humiliated and trampled under the public scourge, has no choice left but to go away and croak.)

The list of perverse effects of anonymous refereeing goes on.

Anonymity encourages suspicion: when receiving a nasty review, it is tempting to start the guessing game; life-long hatreds are born that way, justified or not [1]. 

And one more, possibly the worst. For the truly dishonest, anonymity is a wonderful instrument of plagiarism. How easy! You read your competitor’s latest work, reject it under some pretext (there is always something to criticize, especially in adventurous, leading-edge research), and quickly publish the ideas under your name. The victim may suspect what happened (with more than a guess in this case), but will never be able to prove anything.

The fundamental flaw is the lack of symmetry of the situation. Of the four possible cases shown below, number 2, the currently dominant practice of one-way anonymity, is unfair. The fair possibilities (#3 does not seem to make sense) are the symmetric ones:  #1, full openness, and #4, double-blind refereeing.

While double-blind is better than one-way anonymity, it has unpleasant effects of its own. It is typically not that hard, even if the authors honestly play the game, to unmask them. Playing that game fully means going to great lengths to obscure your identity, and may distort truth; for example, if you are expanding on your own earlier work, you have to pretend instead that you are commenting on someone else’s papers, getting dangerously close to lying. One may ask in the end why researchers should devote their efforts to such silliness rather than to research. Perfecting the obfuscation of your own papers does not advance science. Are there not enough distractions already, from mission statements and grant proposals and financial plans and justifications and reports on how you used the money? Lest we forget, research is about solving problems and writing up your results.

Open refereering [2] is the simple, clear and honest solution. It should be the default. What do we fear? We evaluate a peer’s contribution, saying who we are and what we think of it. A normal day-to-day task of responsible professionals.

Some cases justify anonymity, but they are marginal. Being embarrassed about reviewing someone’s work because of some conflict of interest is not one of them: in such cases, you just decline. In fact, standard professional rules require it. The only critical cases involve foul play. If you discover evidence of fraud, you may prefer to notify the editor anonymously at first. But these are borderline situations. In the work of a scientist, which includes helping the progress of one’s discipline through refereeing, open refereeing is the practical and ethical solution.

Notes

[1] All his life, Einstein believed the reason he did not get a position at ETH was a bad recommendation by Poincaré, who in reality had endorsed him enthusiastically.

[2] Which I have practiced for a long time; see here. (That text is long-winded and I would write it better today, but the content stands.)