SC Turns Reproducibility Challenges Into Opportunities

The SC conference technical program leads the industry in many aspects of computational science delivering ground breaking technical papers, panels, posters and BOFs. Over the life of the conference, the field of computational science has matured, and is now often referred to as the third leg of science (alongside theory and experimentation).

To maintain and build on our leadership, we must develop workflows and practices that ensure robust, repeatable scientific results, even as the platforms we use for computational science continue to evolve. To support this, we propose to institute practices over the next five years that establish SC as the "gold standard" for computational science practice, as measured by the ability of a researcher to reproduce published research findings and validate scientific conclusions from simulations.

There are complex factors affecting reproducibility and replication in computational science, and there are also costs in both time and resources to institute the measures necessary to ensure reproducibility. For this reason, we are soliciting input from the SC technical community. This week, the SC program features multiple events promoting discussion on reproducibility challenges:

• On Tuesday, Nov. 17, we start with with a BoF on "Performance Reproducibility in HPC – Challenges and State-of-the-Art" with Rudi Eigenmann (Purdue University/National Science Foundation USA), Martin Schulz (Lawrence Livermore National Laboratory USA), Torsten Hoefler (Swiss Federal Institute of Technology, Zurich, Switzerland), Georg Hager (University of Erlangen-Nuremberg, Germany), and Sascha Hunold (Technical University of Vienna, Austria) as the panelists. [more here]
• On Wednesday, Nov. 18, SC15 hosts an invited talk on "Reproducibility in High Performance Computing" by Victoria Stodden (University of Illinois at Urbana-Champaign). [more here]
• On Thursday, Nov. 19, we continue with a lunch BoF on "Reproducibility of High Performance Codes and Simulations – Tools, Techniques, Debugging" with Jim Demmel (University of California Berkley), Michael Heroux (Sandia National Laboratory), Ganesh Gopalakrishnan (University of Utah), Walid Keyrouz (U.S. National Institute of Standards and Technology), and Kento Sato (Lawrence Livermore National Laboratory) as panelists [more here],  and a paper on performance reproducibility titled "Scientific Benchmarking of Parallel Computing Systems" in the first afternoon session. [more here]
• On Friday, Nov. 20, the conference ends with the first Workshop on Numerical Reproducibility at Exascale (NRE2015). [more here]

In 2016, we move to actions! The Student Cluster Challenge is being re-designed to focus on the replication of a previously published conference paper. Authors of technical papers accepted at SC15 (and previous years) are invited to submit proposals to reproduce their paper results. Submissions will be accepted soon – from December 1, 2015 to January 15, 2016 – and the selected paper’s software application will be featured as a competition application in the Student Cluster Competition at SC16.

We hope these steps will both improve our understanding of the challenges and raise awareness of the direction we are taking. Please share your thoughts and join us in transforming HPC at the exascale!

Michela Taufer is an associate professor, and holds the David and Beverly J.C. Mills Career Development Chair in the Department of Computer and Information Sciences at the University of Delware. Wilfred Pinhold is CEO of Concurrent Systems LLC and an adjunct professor at Portland State University.