Communications of the ACM
Repeatability in Computer Systems Research
Credit: Anthony Freda
In 2012, when reading a paper from a recent premier computer security conference, we came to believe there is a clever way to defeat the analyses asserted in the paper, and, in order to show this we wrote to the authors (faculty and graduate students in a highly ranked U.S. computer science department) asking for access to their prototype system. We received no response. We thus decided to reimplement the algorithms in the paper but soon encountered obstacles, including a variable used but not defined; a function defined but never used; and a mathematical formula that did not typecheck. We asked the authors for clarification and received a single response: "I unfortunately have few recollections of the work ... "
Key Insights
We next made a formal request to the university for the source code under the broad Open Records Act (ORA) of the authors' home state. The university's legal department responded with: "We have been unable to locate a confirmed instance of [system's] source code on any [university] system."
Comments
CACM Administrator
The following letter was published in the Letters to the Editor of the May 2016 CACM (http://cacm.acm.org/magazines/2016/5/201586).
--CACM Administrator
Christian Collberg and Todd A. Proebsting deserve our gratitude for their article "Repeatability in Computer Systems Research" (Mar. 2016) shining sunlight the best kind of disinfectant, according to Supreme Court Justice Louis D. Brandeis on the very real problem of lack of repeatability in computer science research. Without repeatability, there is no real science, something computer science cannot tolerate.
Alex Simonelis
Montral, Canada
Displaying 1 comment