Sign In

Communications of the ACM

Departments

Reboot the Computing-Research Publication Systems


View as: Print Mobile App ACM Digital Library Full Text (PDF) In the Digital Edition Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
Former CACM Editor-in-Chief Moshe Y. Vardi

A year ago, I proposed that ACM establish a policy change for its conferences, requiring that authors of accepted papers may opt out from in-person involvement and contribute instead by video.a "Be careful what you wish for," says an idiom. By mid-March 2020, conferences were forced to virtualize due to COVID-19. It is now clear that conferences will continue to be virtual at least until the middle, if not the end, of 2021, and perhaps even beyond that. While I am happy this will prevent adding tens of thousands of tons of CO2 to the atmosphere, the virtualization of conferences sharpened my conviction that the computing-research publication system is badly broken and is in need of a serious reboot.

How did we get here? Back in the 1960s, journals were slow, while the field was new and needed to move fast. Conferences offered a solution: Present a preliminary version in a conference with a six-month submit-decide-present cycle, get feedback and credit, and then publish an archival version in a journal. Program committees then did not review papers; they selected papers for the program.

Over time, the selection process became a review process: decision notices became decision notices with some feedback, feedback evolved into reviews, and reviews eventually led to rebuttals. Furthermore, conferences have evolved from being a venue for preliminary publication to, in practice, a venue for archival publication. Also, conference-program time pressure, led to selectivity, which led to prestige. As a result, computing-research conferences today are clearly the preferred venue for archival publication.

But conferences were never designed to provide a venue for high-quality archival publication, since they lack the appropriate editorial process, whose essential element is iterative improvement. It is not uncommon for published journal papers to go through two and even three versions before the reviewers and editors are satisfied. Conferences that run "review experiments," such as NIPS 2014b and ESA 2018,c have concluded there is a huge element of randomness in conference editorial decisions. "The reputation of the peer-review process is tarnished," concluded Hannah Bast. Practically everyone in computing research complains about the "reviewers," but the reviewers are us! If everyone is unhappy, then the problem must be systemic.

Probably the strongest arguments in favor of conferences are their predictability, in terms of submission and decision timing, and their function as venues for community building. But the promised predictability is an illusion, as papers bounce from conference to conferences until they finally find a home. And when every paper has to be published in a conference, conferences are attended mostly by junior researchers, while 20–30 years ago they offered a truly representative sample of their communities, where junior and senior researchers mingled.

A natural reaction to loss is to recreate the familiar. When a conference was "a journal that meets in a hotel," space restriction forced a time restriction, so a conference typically lasts 2–4 days. We kept this tradition in virtual conferences, which still meet for 2–4 days, even though the original reason is gone. But if we have learned anything over these past months it is that spending a day in virtual space is quite difficult and screen fatigue is a real phenomenon. Yet, while we all long for COVID-19 to be over, the climate-change threat is looming larger, and I suspect that virtual conferences are here to stay.

So, we seem to be stuck with a dysfunctional, antiquated publication system. It is time to end the debate about journals and conferences. Let us design a new publication system, something we, as computing professionals, should know how to do. We should collect system requirements, design the system, implement prototypes, experiment, and iterate. The publication system is our system. We are in charge! Technology opens new avenues, but we must be imaginative and not be bound by the dogmas of the dysfunctional past.

If we have learned anything from COVID-19 it is that dealing with major societal challenges requires collective action. The U.S., with its tradition of "rugged individualism" and under meager federal leadership, is handling the pandemic quite poorly. But enabling collective action is exactly why we have established professional societies. They must lead the way.

It is a cliché that everyone wants change, but no one wants to change. Let us collectively agree to change. We deserve a publication system that meets the needs of science, of scientists, and of society. Let us reboot the publication system for computing research. ACM is launching a Presidential Task force on Future Formats for ACM Conferences. It is a start!

Back to Top

Author

Moshe Y. Vardi (vardi@cs.rice.edu) is University Professor and the Karen Ostrum George Distinguished Service Professor in Computational Engineering at Rice University, Houston, TX, USA. He is the former Editor-in-Chief of Communications.

Back to Top

Footnotes

a. https://bit.ly/2KgHHr5

b. https://bit.ly/2UKEhiq

c. https://bit.ly/3lSIqg0


Copyright held by author.
Request permission to (re)publish from the owner/author

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.


Comments


Gunnar Wolf

There is another important and quite interesting side effect for the shift of conferences to online gatherings: Inclusivity. It is now easier than ever before for interested people all over the world to become a conference participant; travel and accommodation costs for attending a USA- or Europe-based conference is oftentimes prohibitive for individuals living in less developed countries, even more if they are practitioners or work at underfunded universities. I truly hope we maintain the ability to participate without the presentiality requirement after the COVID crisis is truly done.
Of course, there are many other positive sides to this effect. While a specific individual might not be interested in taking part of a whole conference on a given subject, being able to "attend" specific lectures or discussions only can also improve across-the-board participation; computing has grown so much as a discipline that we have huge distances between subfields; if I specialize in, say, architectures and operating systems, I might not be motivated to join AI or HCI conferences, and will not benefit my work from their advances. But if I can just grab a chair and attend one or two sessions that are closer to my field -- we can start to re-cross-polinize between computer science fields!


Displaying 1 comment