Sign In

Communications of the ACM

BLOG@CACM

When Reviews Do More Than Sting


View as: Print Mobile App ACM Digital Library Full Text (PDF) In the Digital Edition Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
Bertrand Meyer

http://cacm.acm.org/blogs/blog-cacm/123611-the-nastiness-problem-in-computer-science/fulltext
August 22, 2011

Are we malevolent grumps? Nothing personal, but as a community, computer scientists sometimes seem to succumb to negativism. They admit it themselves. A common complaint in the profession is that instead of taking a cue from our colleagues in more cogently organized fields such as physics, who band together for funds, promotion, and recognition, we are incurably fractious. In committees, for example, we damage everyone's chances by badmouthing colleagues with approaches other than ours. At least this is a widely perceived view ("Circling the wagons and shooting inward," as Greg Andrews put it in a recent discussion). Is it accurate?

One statistic that I have heard cited is that in 1-to-5 evaluations of projects submitted to the U.S. National Science Foundation the average grade of computer science projects is one full point lower than the average for other disciplines. This is secondhand information, however, and I would be interested to know if readers with direct knowledge of the situation can confirm or disprove it.

More examples can be found in the material from a recent keynote by Jeffrey Naughton, full of fascinating insights (see http://pages.cs.wisc.edu/~naughton/naughtonicde.pptx). Naughton, a database expert, mentions that only one paper out of 350 submissions to SIGMOD 2010 received a unanimous "accept" from its referees, and only four had an average accept recommendation. As he writes, "either we all suck or something is broken!"

Much of the other evidence I have seen and heard is anecdotal, but persistent enough to make one wonder if there is something special with us. I am reminded of a committee for a generously funded CS award some time ago, where we came close to not giving the prize at all because we only had "good" proposals, and none that a committee member was willing to die for. The committee did come to its senses, and afterward several members wondered aloud what was the reason for this perfectionism that almost made us waste a great opportunity to reward. We come across such cases so oftenthe research proposal evaluation that gratuitously but lethally states that you have "less than a 10% chance" of reaching your goals, the killer argument "I didn't hear anything that surprised me" after a candidate's talkthat we consider such nastiness normal without asking any more whether it is ethical or helpful. (The "surprise" comment is particularly vicious. Its real purpose is to make its author look smart and knowledgeable about the ways of the world, since he is so hard to surprise; and few people are ready to contradict it: Who wants to admit that he is naïve enough to have been surprised?)

A particular source of evidence is refereeing, as in the SIGMOD example. I keep wondering at the sheer nastiness of referees in CS venues.

We should note that the large number of rejected submissions is not by itself the problem. Naughton complains that researchers spend their entire careers being graded, as if passing exams again and again. Well, I too like acceptance better than rejection, but we have to consider the reality: with acceptance rates in the 8%20% range at good conferences, much refereeing is bound to be negative. Nor can we angelically hope for higher acceptance rates overall; research is a competitive business, and we are evaluated at every step of our careers, whether we like it or not. One could argue that most papers submitted to ICSE and ESEC are pretty reasonable contributions to software engineering, and hence these conferences should accept four out of five submissions; but the only practical consequence would be that some other venue would soon replace ICSE and ESEC as the publication place that matters in software engineering. In reality, rejection remains a frequent occurrence even for established authors.

Rejecting a paper, however, is not the same thing as insulting the author under the convenient cover of anonymity.

The particular combination of incompetence and arrogance that characterizes much of what Naughton calls "bad refereeing" always stings when you are on the receiving end, although after a while it can be retrospectively funny; one day I will publish some of my own inventory collected over the years. As a preview, here are two comments on the first paper I wrote on Eiffel, rejected in 1987 by the IEEE Transactions on Software Engineering (it was later published, thanks to a more enlightened editor, Robert Glass, in the Journal of Systems and Software). The IEEE rejection was on the basis of such review gems as:

  • I think time will show that inheritance (section 1.5.3) is a terrible idea.
  • Systems that do automatic garbage collection and prevent the designer from doing his own memory management are not good systems for industrial-strength software engineering.

One of the reviewers also wrote: "But of course, the bulk of the paper is contained in Part 2, where we are given code fragments showing how well things can be done in Eiffel. I only read 2.1 arrays. After that I could not bring myself to waste the time to read the others."

This is sheer boorishness passing itself off as refereeing. I wonder if editors in other, more established disciplines tolerate such attitudes. I also have the impression that in non-CS journals the editor has more personal leverage. How can the editor of IEEE-TSE have based his decision on such a biased an unprofessional review? Quis custodiet ipsos custodes?

"More established disciplines." Indeed, the usual excuse is that we are still a young field, suffering from adolescent aggressiveness. If so, it may be, as Lance Fortnow has argued in a more general context, "time for computer science to grow up." After some 60 or 70 years we are not so young any more.


Rejecting a paper is not the same thing as insulting the author under the convenient cover of anonymity.


What is your experience? Is the grass greener elsewhere? Are we just like everyone else, or do we truly have a nastiness problem in computer science?

Back to Top

Readers' Comments

This is only a problem for academics. In the real world (industry), the customers stand in judgment.
        Anonymous

I am a physicist but have entered CS and now publish in this field. I do notice the attitudes you describe and they scare me because I get the impression that every other computer scientist is very insecure. Rude comments from reviewers are common and editors seem not to care. But more so, it is common that reviewers are clueless and barely understand the paper they review. So, if one reviewer is rude and clueless, and two are knowledgeable and positive, then the editor still mainly listens to the clueless one, simply because a negative critique is more worthy than positive in this field...
        Anonymous

As a reviewer and as an author, I get the feeling (in some cases I actually know) that some of my (co)reviewers did one of two things: Had someone else less/not qualified review the paper, without bothering to check the quality of the review; or reviewed the paper at the last possible minute, probably after several reminders from the program chair.

In either case, it is hard to get a fair review.
        Anonymous

I am from physics, it has its own share of nastiness, different from what you describe. Right now, I work in a research organization dominated by computer scientists and have written and reviewed some computer science papers. At the risk of sounding haughty (I do not mean to), I would say:

As you have mentioned, computer science is a relatively new field and physics far more mature. This not only means that computer science has more upstarts reviewing and writing papers, but that the quality of research varies from excellent to mediocre to rather poor. As opposed to physics or natural sciences, where almost all research in a field is of similar quality (with respect to maturity). Now you might say that is good or not good, I don't know.

Also, computer scientists have far more funds to publish and hold conferences (at exotic locations), leading in turn to lots more papers to write and review, and all the related rage. I wrote one paper in two years and reviewed maybe a couple every year while I was in physics. I do some reviewing/writing activity every week in computer science research.
        Anonymous

One challenge this poses to program chairs is that we can be misled by nasty reviews versus genuine rejections, especially when the nasty guy works hard. A solution might be to publish reviewer stats, including how often a reviewer is the minority, average length of review, and so on. It might improve behavior if we know that poor numbers could brand us out of prestigious committees. We all want to be on program committees, and then act as if we can't be bothered and are too busy! This needs to change. Coming from industry, I know that sticks work better than carrots!
        Anonymous

Back to Top

Author

Bertrand Meyer is a professor at ETH Zurich and ITMO (St. Petersburg) and chief architect of Eiffel Software.


©2013 ACM  0001-0782/13/02

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from permissions@acm.org or fax (212) 869-0481.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2013 ACM, Inc.


Comments


Anonymous

Publishing reviewer statistics is a great idea and will make reviewers more careful before they provide hasty evaluations. Recently our paper in CHI was rejected because one reviewer gave a strong negative rating without really even justifying his decision. His comments showed he had clearly not read the paper fully. While other reviewers scored our paper 2 categories higher giving favorable reviews. Conferences still resort to average scores and ignore such situations where it is clear that the reviewer has not done his job properly.


Anonymous

I think that the reviewing process also should be more open. I think that reviews should be available together with the accepted papers. This can be done still preserving reviewers anonymity.
This way you also get the average of scores and so on....


Philip Godfrey

Regarding NSF grant proposal scores, apparently CISE (i.e. computer science) proposals average 0.41 points lower than other directorates. Data via Jeannette Wing:
http://cacm.acm.org/blogs/blog-cacm/134743-yes-computer-scientists-are-hypercritical/fulltext


Mark Wallace

"Rejecting a paper is not the same thing as insulting the author under the convenient cover of anonymity."

Perhaps this is a training problem, and reviewers should first be instructed in how to reject papers without being "insulting." Unfortunately, what is or is not an insult is, to a certain extent, in the eye of the beholder. It isn't easy to imagine anyone who worked hard on a paper receiving a rejection notice without some bad feeling, and yet, as Dr. Meyer points out, the vast majority of submissions have to be rejected.


Rafael Anschau

The real problem is that we still dont have a way to measure the quality of research in CS the same way physics has. In physics, if a theory is refutable and survives a few tests, it is considered of quality. A CS thesis simply depends on so many facts that's hard to enumerate. Which criteria the judges of garbage collection should have used to evaluate the idea ? CS research evaluation is still very subjective, so biases become substitute for tests. CS needs its own Karl Popper.


Anonymous

I have served on a variety of panels and program committees and indeed I see that happening often. On the other hand, it is also the job of Program Directors and Program Chairs to filter and guide reviewers to avoid this.


Anonymous

"Rejecting a paper is not the same thing as insulting the author under the convenient cover of anonymity."

This is so true. We understand that the vast majority of submissions have to be rejected. But reviewers MUST contribute to the paper. The systems will judge papers by their grades. Reviewers do not need to judge, they need to review and contribute. Reviewers must provide facts: "authors should have read "; "authors should have employed "; and so on. A good review do not need to be positive. It do not need to approve. It must contribute to the paper!


Anonymous

The author is sadly correct. I have a similar collection of woefully inadequate comments and have been subjected on occasions to an almost sneering approach. One of my better pieces of work (after years of experience, you know which ones they are), had such an unfair battering from IEEE Transactions on Software Engineering, a journal I have published in before,that I complained about the reviewing process and got an acknowledgement but that was all in spite of several follow-up requests. Pathetic. Another to the Journal of Cryptology invited two reviewers. One reviewer said the algorithm was trivially like something else (it wasn't) and the other didn't understand it and said so, even though there was supporting experimental data. Needless to say it was roundly rejected. I'm old enough and experienced enough to shrug it off, but when it happens to a new PhD student, it really undermines them and you have to spend time counselling them about the vagaries of the process. Everybody understands that many papers need rejection or at least significant rethinking but this has to be a constructive and thoughtful process. It all too often isn't.

After years of publishing in a number of these journals, I've mostly given up and publish in ArXiV. I also refuse to review for those journals that behave like this. It really is time for CS to grow up.


Frederick Carlson

The author is correct. ACM is in a league of it's own when it comes to flat out nastiness. The first (and probably last) paper I submitted to an ACM conference had 3 referees. 2 were helpful, the 3rd guy was personally insulting, unprofessional and, worse, unhelpful.


CACM Administrator

The following letter was published in the Letters to the Editor in the April 2013 CACM (http://cacm.acm.org/magazines/2013/4/162502).
--CACM Administrator

Bertrand Meyer's blog post "When Reviews Do More Than Sting" (Feb. 2013) is an opportunity to reflect on how CS academic publishing has evolved since it was first posted at blog@cacm (Aug. 2011). Meyer rightly identified rejection of 80% to 90% of conference submissions as a key source of negative reviewing, with competitors ready to step in with even higher rejection rates, eager to claim the quality mantle for themselves.

In recent years, we have seen that conference quality can be improved and constructive reviewing facilitated, even when a greater proportion of papers is accepted. At least six conferences, including ACM's Special Interest Group on Management of Data (SIGMOD), Computer-Supported Cooperative Work (CSCW), and High Performance Embedded Architectures and Compilers (HiPEAC), incorporate one or more paper-revision cycles, leading to initial reviews that are constructive rather than focused on grounds for rejection. Giving authors an opportunity to revise also provides a path toward accepting more submissions while still improving overall conference quality.

Analyses by Tom Anderson of the University of Washington and George Danezis of Microsoft Research suggest there is little or no objective difference among conference submissions that reviewers rank in the top 10% to 50%. Many conferences could even double their acceptance rates without diminishing their quality significantly, even as a serious revision cycle would improve quality.

This change in the CS conference process would blend conference and journal practices. Though journal reviews may not always be measured and constructive, on balance they are, and, in any case, revision cycles are a way for conferences to be more collegial.

Jonathan Grudin
Redmond, WA


Displaying all 10 comments