How Should Peer Review Evolve?

Peer review publications have been around scientific academic scholarship since 1665, when the Royal Society’s funding editor Henry Oldenburg created the first scientific journal. As Jeannette Wing nicely argued in her recent blog post here, it is the public, formal, and final archival nature of the process of the Oldenburg model that established the importance of publications to scientific authors, as well as their academic standings and careers.  

Recently, as the speed of communication of research results reach breakneck speeds, some have argued that it is time to fundamentally examine the peer review model, and perhaps to modify it somewhat to suit the modern times.  One such proposal recently posed to me via email is Open Peer Review, a model not entirely unlike the Wikipedia editing model in many ways.  Astute readers will realize the irony of how it is the Wikipedia editing models that often make academics squirm in their seats.

The proposal for open peer review suggests that the incumbent peer review process has problems in bias, suppression, and control by elites against competing non-mainstream theories, models, and methodologies.  By opening up the system, we might increase accountability and transparency of the process, and mitigate other flaws.  Unfortunately, while we have anecdotal evidence of these issues, there remains significant problems in quantifying these flaws with hard numbers and data, since reviews often remain confidential.

Perhaps more distressing is that several experiments in open peer review (such as done by the journal Nature in 2006, and British Medical Journal in 1999, Journal of Interactive Media in Education in 1996) have had mixed results in terms of the quality and tone of the reviews.  Interestingly, and perhaps unsurprisingly, many of those who are invited to review under the new model decline to do so, potentially reducing the pool of reviewers.  This is particularly worrisome for the academic conferences and journals, at a time when we desperately need more reviewers due to the growth of the number of submissions.

A competing proposal might be open peer commentary, which elicit and publish commentary on peer-reviewed articles.  This can be done prior to publication, or even after the date of publication.  In fact, recent SIGCHI conferences have already started experimenting with this idea, with several popular paper panels in which papers are first presented, and opinions from a panel is openly discussed with an audience.  The primary focus here is to increase participation, which might also improve transparency.  The idea of an open debate, with improved transparency, is of course the corner stone of Wikipedia editing model (and our research project WikiDashboard).

Finally, it is worth pointing out the context under which these proposals might be evaluated.  We live in a different time than Oldenburg. In the mean time, communication technology have already experienced several revolutionaries of gigantic proportions. Now, real-time research results are often distributed, blogged, tweeted, facebooked, googled, and discussed in virtual meetings.  As researchers, we can ill afford to stare at these changes and not respond.  

Beyond fixing problems and issues of bias, suppression, and transparency, we also need to be vigilant of the speed of innovation and whether our publication processes can keep up.  Web review management systems like PrecisionConference have gone a long ways in scaling up the peer-review process.  What else can we do to respond to this speed of growth yet remain true to the openness and quality of research?