Sign In

Communications of the ACM

Viewpoint

The Real Software Crisis: Repeatability as a Core Value


The Real Software Crisis, illustration

Where is the software in programming language research? In our field, software artifacts play a central role: they are the embodiments of our ideas and contributions. Yet when we publish, we are evaluated on our ability to describe informally those artifacts in prose. Often, such prose gives only a partial, and sometimes overly rosy, view of the work. Especially so when the object of discourse is made up of tens of thousands of lines of code that interact in subtle ways with different parts of the software and hardware stack on which it is deployed. Over the years, our community's culture has evolved to value originality above everything else, and our main conferences and journalsa deny software its rightful place.

Science advances faster when we can build on existing results, and when new ideas can easily be measured against the state of the art. This is exceedingly difficult in an environment that does not reward the production of reusable software artifacts. Our goal is to get to the point where any published idea that has been evaluated, measured, or benchmarked is accompanied by the artifact that embodies it. Just as formal results are increasingly expected to come with mechanized proofs, empirical results should come with code.


Comments


Sven Apel

Hi Shriram & Jan,

Thanks for your insights, its a great article!

Interestingly, I noticed a connection to the problem of encouraging,
conducting, and evaluating replication studies in software-engineering research.
Based on a survey involving 79 PC and editorial-board members of major
software-engineering venues, we found that there is a deep confusion and
disagreement in the community regarding the value and guidelines of
empirical studies, in general, and replication, in particular.

http://www.infosun.fim.uni-passau.de/cl/publications/docs/SiSiAp15.pdf

The situation is not unlike the situation you outline...

Best,

Sven


Displaying 1 comment

Log in to Read the Full Article

Sign In

Sign in using your ACM Web Account username and password to access premium content if you are an ACM member, Communications subscriber or Digital Library subscriber.

Need Access?

Please select one of the options below for access to premium content and features.

Create a Web Account

If you are already an ACM member, Communications subscriber, or Digital Library subscriber, please set up a web account to access premium content on this site.

Join the ACM

Become a member to take full advantage of ACM's outstanding computing information resources, networking opportunities, and other benefits.
  

Subscribe to Communications of the ACM Magazine

Get full access to 50+ years of CACM content and receive the print version of the magazine monthly.

Purchase the Article

Non-members can purchase this article or a copy of the magazine in which it appears.