Sign In

Communications of the ACM

Contributed articles

Repeatability in Computer Systems Research

Repeatability in Computer Systems Research, illustration

Credit: Anthony Freda

In 2012, when reading a paper from a recent premier computer security conference, we came to believe there is a clever way to defeat the analyses asserted in the paper, and, in order to show this we wrote to the authors (faculty and graduate students in a highly ranked U.S. computer science department) asking for access to their prototype system. We received no response. We thus decided to reimplement the algorithms in the paper but soon encountered obstacles, including a variable used but not defined; a function defined but never used; and a mathematical formula that did not typecheck. We asked the authors for clarification and received a single response: "I unfortunately have few recollections of the work ... "

Back to Top

Key Insights


We next made a formal request to the university for the source code under the broad Open Records Act (ORA) of the authors' home state. The university's legal department responded with: "We have been unable to locate a confirmed instance of [system's] source code on any [university] system."


CACM Administrator

The following letter was published in the Letters to the Editor of the May 2016 CACM (
--CACM Administrator

Christian Collberg and Todd A. Proebsting deserve our gratitude for their article "Repeatability in Computer Systems Research" (Mar. 2016) shining sunlight the best kind of disinfectant, according to Supreme Court Justice Louis D. Brandeis on the very real problem of lack of repeatability in computer science research. Without repeatability, there is no real science, something computer science cannot tolerate.

Alex Simonelis
Montral, Canada

Displaying 1 comment

Log in to Read the Full Article

Sign In

Sign in using your ACM Web Account username and password to access premium content if you are an ACM member, Communications subscriber or Digital Library subscriber.

Need Access?

Please select one of the options below for access to premium content and features.

Create a Web Account

If you are already an ACM member, Communications subscriber, or Digital Library subscriber, please set up a web account to access premium content on this site.

Join the ACM

Become a member to take full advantage of ACM's outstanding computing information resources, networking opportunities, and other benefits.

Subscribe to Communications of the ACM Magazine

Get full access to 50+ years of CACM content and receive the print version of the magazine monthly.

Purchase the Article

Non-members can purchase this article or a copy of the magazine in which it appears.