Yes, Computer Scientists Are Hypercritical

Another way to address the problem is to change the system so that reviewers do not provide absolute ratings of proposals, but rather provide comparative rankings of proposals. This eliminates the effect of each reviewer's general tendency towards hypo- or hyper-criticality

I wrote a paper on this very piont a few years ago:
http://research.microsoft.com/apps/pubs/default.aspx?id=80226

John Douceur
Microsoft Research

I can't believe the shoddy statistical analysis passed off here as "scientific research". In this day and age it is astounding that someone would show only averages for CISE and non-CISE, without doing a Student's t test or Mann-Whitney-Wilcoxon test to see whether the result is statistically significant. The conclusion validity of the work is almost nil. Maybe if we reject this blog post, the author will get the message and do better work in her next post.

--Jamie Andrews
University of Western Ontario
(yes, I am joking)

what is the distribution / curves for the other non-CISE disciplines? is CISE really that much of an outlier, or is there quite a range?

What about the hypothesis that CISE proposals actually deserve these ratings? We can't eliminate this just because it would not be convenient for us.

Doesn't it just show that comparing numeric scores of reviewers across disciplines and even within a program committee meeting is pointless?

I would also like to know how the other research scored. Is CISE an outlier?

One of the main causes for this is the CS emphasis on highly selective conferences. In CS, everything you ever do is subject to the filter of fierce competition. Everything is a zero-sum game, and everyone ends up having good papers rejected. Since authors are also reviewers, the acceptance threshold is continuously going up. Eventually, only outstanding contributions are good enough, and even the significance of those can be fiercely debated at program commitee meetings. Obviously, this spills over to proposal evaluation. The only way to eliminate this problem is to abolished these highly selective conferences. Science has worked great without them for 500 years, they are doing more harm than good. Let conferences be conferences (i.e. a place to just talk about stuff) and rely on journals for high selectivity. The journal reviewing process does not have the problems of conference reviewing.

My hypothesis is that 10-20 years ago Computer Science researchers only had to compete against other CS researchers, generally for targeted funding. Being "hypercritical" of other research meant your research (or your subdiscipline, or your collaborators/friends) was more likely to get funding. These days, CS no longer has targeted funding, rather it's just another discipline in the general science & engineering funding mix.
Clearly our learned behavior is no longer appropriate in this context, when we are competing against a much wider pool.

The only way to eliminate this problem is to abolished these highly selective conferences... The journal reviewing process does not have the problems of conference reviewing.

Oh come on. Inasmuch as the "problem" is selectivity, then highly selective journals are just as bad as highly selective conferences. Talk to Physicists about Letters to Nature, or PRLs, about the difficulty of refereeing on a one week deadline, about squeezing references down so their paper fits on two or four pages with citations being nothing but first author, journal, page number --- and as a result, the active researchers look far more to informal web "archives" than anything a librarian would consider "archival".

I don't think researchers are any less competitive in Physics or Biology (or English Lit. for that matter). The difference is that other disciplines have decades of experience in "fighting their corner", whereas for whatever reason, we Computer Scientists act as if that is beneath us.

For better or for worse, Computer Science came of age after the 747 - and our conference based-publication structure reflects that as much as anything.

I'm very happy to see the data reported as "hypercriticality" instead of nastiness. We are indeed hypercritical, and it's perfectly explainable.

Conjecture: there is no nastiness problem, just a confluence of two inherent factors. First, we are an area that focuses (more than others) on conceptual novelty. Second, there is very rarely something conceptually new in CS, especially in applied directions and especially at the proposal (and not paper) level.

In support of the conjecture:
-I've seen a lot of Physics reviews. (My wife is a physicist.) They are by no means more "professional" than in CS. If nothing else, they are nastier.

-Our area itself has trained us to analyze everything and focus on conceptual novelty. We have thinking processes that let us distill everything and then compose it in complex ways. "Software is distilled complexity", etc.

-It is really hard to have conceptual novelty in most areas of computer science. Even the most celebrated ideas are similar at the high-level to something that has been done before. The interesting "twist" is always low-level. For proposals, this is deadly, because there can be no proof (e.g., experiments or new algorithms) that what one does is indeed new. People have to rely on high-level intuitions. This is not the case in other sciences. When one says "I will use this kind of biological mechanism to explain/influence this kind of real-life phenomenon", there is no doubt about conceptual novelty. Either this has been done before or not. The only question is whether it's believable that it is feasible and whether the PI is the best person to do it.