Reflections on Computer-Related Risks

Computer-related technologies have changed enormously over the years, with huge advances in processor power and storage capacity, high-speed networking, and highly distributed systems. Client-server and virtual-machine architectures seem to be simplifying implementation. Internet browsers have significantly raised the level of abstraction for attaining almost universal interoperability. Strong cryptography has become more widely available, and is becoming easier to use. Improvements in static-analysis tools and formal methods are having visible results. Many ACM members have been instrumental in some wonderful advances, and have been involved in important technological and social activities. For example, Parnas, Dijkstra, Hoare, Wirth, and many others contributed to system architectures and programming practice. We have also experienced significant advances in networking, graphics, and many other crucial areas.

Conversely, trustworthiness of operating systems and application software is generally poor, particularly with respect to critical requirements such as security, reliability, survivability, evolvability, maintainability, interoperability, and predictable upgradability. Common flaws keep recurring—buffer overflows, faulty bounds checks, and so on. Denial-of-service attacks are easily created and deployed, but largely lacking adequate defenses. Strong cryptography is difficult to embed securely in systems and applications. Software engineering is still more of an ideal concept rather than a disciplined practice; its principled precepts seem to be widely ignored. The Internet has amplified the risks, and seems to encourage various spams, scams, and spoofs. Trustworthiness and particularly security are often not adequately recognized as essential elements—especially in system architectures, system development, and in curricula. The attackers seem to be gaining faster than the defenders. The current state of the practice in the use of computer systems in elections is particularly appalling; the standards are weak, and the bar is set way below the financial sector and even gambling machines. Vulnerabilities in our critical infrastructures are equally worrisome. High-assurance multilevel security is still more or less a dream, although its practical existence in mainstream systems would provide possibilities that do not exist today.

These and many more subjects have been considered in ACM’s SIGSOFT Software Engineering Notes (SEN) (which I created in 1976 and edited until Will Tracz took over in 1994; www. sigsoft.org/SEN/), the ACM Risks Forum (since 1985; www.risks.org), and CACM’s "Inside Risks" columns (since July 1990; www.csl.sri.com/neumann/insiderisks. html). (My book Computer-Related Risks, published in 1995, is still basically sound despite its age, because many things have not fundamentally changed.) Thus, it seems useful to provide some background that might not be familiar especially to younger ACM members, and to consider what lessons might be learned therefrom.

SEN has served as an outlet for discussions of systems that did not work as expected, as well as how such problems might be avoided. But that was perhaps only preaching to the converted. For several years, SEN included an annual updated list of Illustrative Risks to the Public in the Use of Computers—until the list became too long and became searchable online (see www.csl.sri.com/neumann/illustrative.html). With the ever-increasing volume of salient RISKS cases, I am less inclined to keep the index current. Besides, Lindsay Marshall has provided a nice searchable Web site at Newcastle University for the complete RISKS archives (risks.org).

Risky problems are as great today as they were when we first set out to expose and eradicate them.

Over those early years, there was considerable debate within the ACM Council about ACM’s role in representing real-world concerns regarding the use of computers. The discussions within the Council that inspired the establishment of the ACM Risks Forum are described at length in the message from ACM’s president at the time, Adele Goldberg, in the February 1985 issue of CACM. This was placed under the aegis of the ACM Committee on Computers and Public Policy (CCPP), the chairmanship of which I then inherited from Dan McCracken. ACM thereby demonstrated a genuine recognition of the importance of the social implications of our technologies.

The first RISKS issue on August 1, 1985 (see catless.ncl.ac.uk/Risks/1.01.html) includes a summary of Adele Goldberg’s message with an excerpt of the charter, an agenda for the future, a summary of some of the incidents known at the time culled from SEN (which grew into the Illustrative Risks index), items on the strategic defense initiative and Dave Parnas’s resignation from the antimissile defense advisory group, a pointer to Herb Lin’s analysis of that software, a minireview by Peter Denning, and a note from Jim Horning.

Five years after that, CACM Editor-in-Chief Peter Denning and others urged me to establish the monthly column that became "Inside Risks." I am enormously indebted to the members of CCPP—which then included Denning, Parnas, Horning, Nancy Leveson, Jerry Saltzer, and others—who have served as an astute expert review panel for each succeeding would-be column and provided wise counsel on other issues as well.

The overwhelming conclusion from this body of material is that the risky problems are as great today as they were when we first set out to expose and eradicate them. Although the prevention mechanisms have improved somewhat, it is evident that we have not been advancing sufficiently rapidly in the development of mass-marketplace systems and custom applications that are sufficiently trustworthy—despite the tangible gains and research advances I noted in the first paragraph of this essay. Worse yet, various factors have outpaced those mechanisms, including increased complexity of systems, increased worldwide dependence on information technology and the ever-growing Internet, increasingly critical applications to which that technology is being entrusted, the general ease with which antisocial acts can be committed, and the ubiquity of potential attackers. Thus, we seem to be falling farther behind as time goes by. In particular, the huge expansion in the scope and pervasiveness of the Internet is creating many challenges for our community.

One of the biggest challenges for ACM members and for the computer community as a whole is bridging the gap between research and development, and the gap between theory and practice. Clearly, we need to devote greater attention to improving development practices.

In its first 50 years, CACM has been a useful product of the Association for Computing Machinery. However, in the next 50 years, the ACM needs to become—both in spirit and in reality—something more like the Association for Computing Methods or perhaps Methodologists, stressing the vital role of people in the urgent pursuit of transforming computer system development into a true engineering discipline that makes optimal use of the advances of the past 50 years in the context of critical system applications that use the resulting systems wisely. In particular, dramatic changes are needed in developing trustworthy systems that are explicitly designed for human usability for all users, and that encourage well-informed people to take on appropriate responsibilities in environments in which it is clearly unwise to trust technology blindly. For example, see the recommendations of the National Research Council study relating to secure systems and networks, summarized in the October 2007 CACM "Inside Risks" column and the columns relating to the needs for total-system understanding, education, and consistent application of good system-oriented principles.

In 1954, Norbert Wiener wrote about the use of human beings in the context of what he foresaw as the future of computer systems. In 2008, we need to remember that although ACM seeks to improve computer-related technologies and their applications, the purpose of that technology is ultimately to improve the quality of the life for everyone on our planet.

Back to Top