Sign In

Communications of the ACM

BLOG@CACM

Why Peer Review Matters


Carnegie Mellon Professor Jeannette Wing

At the most recent Snowbird conference, where all the chairs of computer science departments in the U.S. meet every two years, there was a plenary session during which the panelists and audience discussed the peer review processes in computing research, especially as they pertain to a related debate on conferences versus journals. It’s good to go back to first principles to see why peer review matters, to inform how we then would think about process.

In research we are interested in discovering new knowledge. With new knowledge we push the frontiers of the field. It is through excellence in research that we advance our field, keeping it vibrant, exciting, and relevant. How is excellence determined? We rely on experts to distinguish new results from previously known, correct results from incorrect, relevant problems from irrelevant, significant results from insignificant, interesting results from dull, the proper use of scientific methods from being sloppy, and so on. We call these experts our peers. Their/our judgment assesses the quality and value of the research we produce. It is important for advancing our field to ensure we do high-quality work. That’s why peer review matters.

In science, peer review matters not just for scientific truth, but, in the broader context, for society’s perception of science. Peer review matters for the integrity of science. Scientific integrity is the basis for public trust in us, in our results, in science. Most people don’t understand the technical details of a scientific result, let alone how it was obtained, what assumptions were made, in what contexts the result is applicable, or what practical implications it has. When they read in the news that “Scientists state X,” there is an immediate trust that “X” is true. They know that science uses peer review to vet results before they are published. They trust this process to work. It is important for us, as scientists, not to lose the public trust in science. That’s why peer review matters.

“Public” includes policymakers. Most government executives and congressional members are not scientists. They do not understand science, so they need to rely on the judgment of experts to determine scientific truth and how to interpret scientific results. We want policymakers in the administration and Congress to base policy decisions on facts, on evidence, and on data. So it is important for policymakers that, to the best of our ability, we, as scientists, publish results that are correct. That’s why peer review matters.

09072010_Peer slide  Jeannette M. Wing
While I argue peer review matters, it’s a whole other question of what the best process is for carrying out peer review. In this day and age of collective intelligence through social networks, we should think creatively about how to harness our own technology to supplement or supplant the traditional means used by journals, conferences, and funding agencies. Peer review matters, and now is the time to revisit our processes--not just procedures and mechanisms, but what it is we review (papers, data, software, and tools), our evaluation criteria, and our incentives for active participation.
 


Read CACM in a free mobile app!
Access the latest issue, plus archived issues and more
ACM Logo
  • ACM CACM apps available for iPad, iPhone and iPod Touch, and Android platforms
  • ACM Digital Library apps available for iOS, Android, and Windows devices
  • Download an app and sign in to it with your ACM Web Account
Find the app for your mobile device
ACM DL Logo