I subscribe to "The Morning Paper," a daily summary prepared by one person, Adrian Colyer, who curates research papers and sends them out to interested readers (https://blog.acolyer.org). Last fall he reviewed "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed DataIntensive Systems" (https://blog.acolyer.org/2016/10/06/simple-testing-can-prevent-most-critical-failures/). It had some surprising results, including:
- Almost all catastrophic failures (48 in total, or 92%) are the result of incorrect handling of nonfatal errors explicitly signaled in software;
- Error handlers with TODO or FIXME in the comments. This example took down a 4,000-node production cluster; and
- Error handlers that catch an abstract exception type (for example, Exception or Throwable in Java) and then take drastic action such as aborting the system. This example brought down a whole Hadoop Distributed File System (HDFS) cluster.
And the list went on from there.
No entries found
Log in to Read the Full Article
Please select one of the options below for access to premium content and features.
Create a Web Account
If you are already an ACM member, Communications subscriber, or Digital Library subscriber, please set up a web account to access premium content on this site.
Join the ACM
Become a member to take full advantage of ACM's outstanding computing information resources, networking opportunities, and other benefits.
Subscribe to Communications of the ACM Magazine
Get full access to 50+ years of CACM content and receive the print version of the magazine monthly.
Purchase the Article
Non-members can purchase this article or a copy of the magazine in which it appears.