SC16 Adds Computational Reproducibility to Student Competition

Data sets and software are important by-products products of research in fields that depend upon data-intensive and high performance computing. But these elements are typically absent when research results are recorded in a journal article or conference proceedings. There is a growing sense in the computational community that this gap needs to be filled if we are to create a stable base of research upon which reliable advances may be built. In short, we need to ensure that computational results are as reproducible as those from experiments.

I am part of a small team at SC16 that over the next several years will work to promote and support replication and reproducibility of computational results. One of the approaches we are exploring is to use SC's Student Cluster Competition (SCC) to test the reproducibility of results presented in papers at previous SC conferences. This is the first time that students have been challenged to reproduce results from a recent computational paper rather than run more traditional benchmark applications.

After evaluating submissions from past SC paper authors, the SCC selected "A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications" by Flick, Jain, Pan, and Aluru for the inaugural reproducibility initiative at SC16 (the paper is available from the ACM Digital Library). Results from this paper will be reproduced by the 14 teams taking part in the cluster competition in November, and one of the paper's co-authors, Chirag Jain, will be helping the SCC committee to create a challenging competition task for the students and will serve as a judge during the competition.

We hope our student teams will be especially engaged by the idea that they aren't running just a sterile benchmark application but rather, by attempting to reproduce previously reported application performance, they are actively taking part in the scientific process. We believe this kind of early engagement will be very important to efforts to make reproducibility a standard part of the computational research process.

Michela Taufer is the David and Beverly J.C. Mills Career Development Chair at the University of Delaware and a member of the SC16 organizing committee.