Conferences vs. Journals in Computing Research

I'm afraid the editorial is completely one-sided and I'm seriously concerned about the ability of Communications to be a good forum "in which to continue this conversation", when its E-i-C has clearly made up his mind.

For the record, other than for areas with involved theoretical results (for which I am prepared to consider that conference reviewing may be too quick), I believe that conference reviewing is not just "careful" but extremely high-quality: by top people and in direct contrast with other submissions for quality and appeal. Considering the quality of the top conferences in my field (programming languages and software engineering) and contrasting it to the average quality of the top journals, I cannot imagine many researchers with a sense of taste preferring the latter. (This is an average-quality assessment and it is not to say that very high quality papers do not also appear in journals and sometimes only in journals.)

I also find it sad that at the exact moment when all other sciences are trying to deal with the inadequacies of their procedures (arXiv is a telltale sign) we bring back a decades-old, fully-resolved question. Conference publication combines a very high-quality vetting mechanism with a forum for quick dissemination of ideas and community forming. It is also a means that easily accommodates industry researchers and practitioners (whose world is not centered around "promotion and tenure committees"). In short, it is a publication mechanism for the 21st century.

Finally, there is nothing wrong with a field setting its own standards for scholarship. The humanities have single-authored books. The sciences have journals. We have a mixture of conferences and journals.

If the problem is "program committee overload", as the Birman and Schneider article suggests, perhaps we should indeed be looking to Physics for solving it: Most Physics papers (e.g., in the Physical Review A-E series of journals, which is the most standard full-length paper venue in Physics) are reviewed by just a single reviewer. Our conference "program committee" overload is due to assigning 3 or more reviewers per paper.

I could not agree more, in fact I had been toying with the idea to submit an opinion piece on this very topic to CACM.

As program chair of an ACM conference I can attest to the many flaws of the conference review process, in spite of everyone's best efforts. Time constraints, concentrated load, difficulty of holding a relaxed rebuttal phase, competition among authors and referees, pressure to go with average ratings, financial concerns (attendance), and other factors --- all bias and weaken the process in important ways.

And as an interdisciplinary researcher, I experience first-hand how our conference-driven publication practices hurt us in terms of impact, reach, and visibility. Computing journals have very low impact factors and very long dissemination times, compared to other science disciplines. Conferences hurt the viability of journals because top researchers are busy refereeing conference papers. The time constraints of conferences make us submit papers in a rush before they are ready, or impose long delays. And competition for the top conferences, as has been noted by others, means that many good papers do not get the attention they deserve.

I propose a simple solution: abolish conference proceedings. Then papers will be submitted to journals instead. Journals will receive more and better papers. Refereeing resources will shift naturally from conferences to journals. As a result, journals will gain impact, improve quality, and speed up their processes. With our full attention, they will become viable once again and the review process will be more rigorous, effective and timely. Deadlines will no longer be concentrated and we can submit better work, revise it until it is ready, and profit immediately and directly from reviewer's feedback -- the same referee can judge improvements to a paper. I could go on but it seems the many advantages are obvious.

We would still hold conference, of course. In many cases where conferences and journals are nicely aligned, presentations can be selected and invited among the best papers published in the previous year. For newer areas and groundbreaking work, a conference or workshop can still accept submissions --- but it should not publish proceedings; publication is the job of journals.

ACM should take the lead in such a transition because it publishes the proceedings of most top computing conferences, as well as many of the top computing journals. Therefore ACM has everything to gain from leading the way. ACM is also the only body that could successfully shepherd such an undertaking.

The switch would not be easy, but with careful planning we could manage a phased transition over a few years and catch up to the rest of the scientific community.

The key is not in Conference vs. Journals. The thing I don't really understand is that why those reviewers (editors, etc.) need so much time (over months) to review a 20-page journal paper. If their time is so valuable for other things, they should not accept the reviewer positions.

When a paper can be timely reviewed, it really doesn't matter it's a conference or journal paper, if it's accepted.

Just because the EiC has an opinion, does not mean that Communications cannot be a forum for a continued conversation. Communications does not publish only opinions that the EiC agrees with. Communication does publish well-reasoned and well-argued opinions.

This is a difficult issue, with no clear-cut line between the two "sides", in my opinion. After submitting papers to journals only to have the results looking decidedly dated when the papers were eventually published, I have moved to a higher proportion of conference papers.

On the issue of subsequent journal publication of conference papers, a practical problem is that many conferences insist on retaining copyright on the paper, and many journals have a requirement that the research should not have been published before. The tensions here are obvious. Otherwise, I would agree that this two-phase process gives the advantages of rapid dissemination and good, archival publication.

One last comment: in my experience, the quality of conference reviewing procedures is of mixed quality, depending on the conference. A good conference has good reviewing procedures (and probably fairly high rejection rates), others are of dubious quality. I'm starting more and more to look at who stands behind a conference when filtering the flood of CFPs received in my email. If a conference has ACM or IEEE backing in some form, my confidence level is greatly increased.

I would like to add my experience as a researcher in bibliometrics. While in other sciences journals are the main or only way to disseminate research findings, a distinctive feature of computer science research is the importance of selective conferences. This peculiarity of research in computer science complicates the process of bibliometric evaluation for two reasons. First, we need to convince other scientists that conference papers are valuable research outcomes and need to be considered in the research evaluation process. Even when this step has been accomplished, however, the problem is how to evaluate conference papers: the two major bibliometric data sources (Thomson Web of Science and Elsevier Scopus) do not index conference papers (only recently the first has launched a conference proceedings index). Thus, bibliometric indicators computed on these data sources usually do not consider conference publications and the resulting scores for computer scientists with a significant share of conference papers (not published in journals) are lower than expected. Google Scholar includes everything but consistency and accuracy of its output is admittedly lower compared to the mentioned commercial citation-enhanced databases. As a result, even when the importance of conference papers is recognized, I experienced non-trivial problems in the bibliometric evaluation of this source of publication.

Several blogs followed up on this topic:

http://blog.computationalcomplexity.org/2009/04/vardi-are-conferences-worth-fixing.html

http://agtb.wordpress.com/2009/04/25/on-vardis-question-on-conferences/

http://jdupuis.blogspot.com/2009/04/conferences-vs-journals-in-computing.html

See also Dan Reed's "Publishing Quarks: Considering Our Culture" at
http://www.cra.org/CRN/articles/march09/Reed-Musings.html