Are Software Developers Affected by Cognitive biases while performing changes to design artifacts? If so, how can we mitigate the impact of such biases on developers' performance? Despite the criticality of these questions, they have been relatively unexplored in the context of software development. When making changes to software artifacts, developers use heuristics that are influenced by cognitive biases. The goal of this article is to examine this influence and to explore whether such biases can be mitigated by using cognitive aids such as traceability (See accompanying sidebar). This article describes the findings of our study and provides recommendations for practice.
People employ cognitive biases to reduce uncertainty and complexity when processing information during problem solving.11 While the effects of these biases have been well studied in general problem-solving, their impact on software development has gained limited attention. These biases are often employed to simplify complex inference tasks to more manageable proportions.8 The most common cognitive biases experienced during problem-solving include anchoring and adjustment, availability, and confirmation.
Anchoring and Adjustment Bias
This bias results from people forming initial estimates about a problem and adjusting their initial estimates to arrive at more appropriate final solutions.8,10 This initial estimate is called as an anchor - 'a familiar/known position or a reference point', that is subject to adjustments.5,7 Typically, these adjustments are insufficient.11 Anchoring and adjustment have been known to impact tasks like artifact reuse and decision making.8 Anchors can be self-generated or externally-generated. To form self-generated anchors, developers retrieve their long-held beliefs (say, past development and domain experiences) from their memory to draw inferences about a task. This is similar to the concept of conservatism. Conservatism is defined based on the following three aspects:
Self-generated anchors exhibit only the second characteristic of conservatism (the long time interval). The other two aspects of conservatism are not necessarily manifested by self-generated anchors.a This discussion implies that conservatism and anchoring are analogous but distinct concepts.
Externally-generated anchors, on the other hand, are triggered by certain artifacts such as system documentation, requirements specification, etc. used by developers. Developers may combine self- and externally-generated anchors or use externally-generated anchors to adjust self-generated anchors to arrive at acceptable solutions. Irrespective of the source, these anchors may be inaccurate and developers may be unaware of these inaccuracies and thereby not realize the need for adjustment. Specific mechanisms are needed to make developers aware of the same. Insufficient adjustments are made when there is an uncertainty about the correct solution. This may lead to premature termination of the task due to reluctance to spend more effort to adjust sufficiently or due to lack of appropriate cognitive resources to facilitate the process of adjustment.
This bias suggests that people are influenced by information that is easier to recall and are prejudiced by information that is well-publicized or recent.10 This suggests that in the absence of any usable knowledge about a system, developers may draw heavily upon their conveniently accessible past experiences. Such experiences may not always be consistent with the current system state and might have undesirable consequences. For example, it might result in erroneous changes to the system if their experiential beliefs are inconsistent with current system state.
This bias suggests that people tend to focus on information that is consistent with their preconceived notions while they disregard information that is inconsistent with their past conceptualizations.3 In other words, people mostly seek evidence that qualifies their beliefs rather than evidence that disqualifies them.1 Developers' past beliefs might dictate the importance placed on new information about requirements, design, and solution.
Since the evidence of the impact of these biases in software development is largely anecdotal, we empirically investigated this phenomenon and the role of traceability in shaping these biases.
Findings from our study (see accompanying sidebar) demonstrate the presence of cognitive biases in software development. Anchoring and adjustment was found to occur at various stages during the process of making changes to system design. Developers alternated between anchoring and adjustment for different purposes (for example, to develop an initial understanding of requirements, system design, and task). Confirmation and availability biases were found to affect the process of anchoring and adjustment through the nature of knowledge available to and sought by the developers.
During each step in the process of handling a change, developers anchored to their initial understanding of the requirement (requirement anchor), current design artifacts (design anchor), and the solution for the change requested (solution anchor). For example, after developers read new requirement specification, they developed an initial understanding (requirement anchor) about the requirement even before actually examining system documentation to understand it in the context of the system. This is typically a combination of self and externally-generated anchors: self-generated from their preconceived notions about the domain and externally-generated by the implied semantics in new requirement specification.2 b
Impact of Traceability
Based on the verbal protocol analysis, we found that developers in the pure control group typically formed anchors that were different and often less accurate when compared to those in the treatment group. Let us consider one of the four maintenance tasks performed by the pure control group: "incorporation of reservation functionality in the library management system." They formed an initial anchor that 'reservation functionality' is same as 'hold functionality' (requirement anchor). This was primarily triggered due to the disuse of traceabil-ity knowledge that could have helped them make the distinction between their belief and the actual functionality. Their interpretation was based solely on their level of familiarity and their mental model of library domain. They had no mechanisms to confirm or disconfirm their beliefs leading them to confide in their incomplete and inconsistent knowledge. They anchored to these estimates and made ill-informed changes that were inaccurate, incomplete, and inconsistent with the current system state. Although subjects in the control and pure control groups did not arrive at incorrect interpretations in every instance (say, when developers needed to understand design of the existing system and the impact of changes), they did so in most instances.
Developers in the treatment group formed more appropriate understanding of current design artifacts (design anchor) and the impact of the changes (solution anchor) more often than the control and pure control groups. For example, they used traceability knowledge to identify the impact of the introduction of the new functionality. Most often, they were also able to correctly identify alternatives available in current design that can be leveraged in implementing the new requirement as a result of the links provided by the traceability knowledge from related requirements to such alternatives. Developers in the control and pure control groups were usually unable to completely identify the impact of the 'reservation functionality' and made decisions based on their initial anchor of what needs to be changed. This observation of differences based on developers' verbal protocols was consistent in most of the instances when subjects needed to understand existing system and the impact of changes. This behavior of developers corroborates the existence of availability and confirmation biases when anchoring to estimates of a solution and the role of traceability in mitigating the effect of these biases.
Developers iteratively adjusted their anchors as they navigated through various design artifacts. After examining a few design artifacts and without expending significant cognitive effort in understanding what might be relevant to the change request, developers initially formed anchors about the current system state. However, developers did not always adhere to these anchors. As they discovered new information from the design artifacts and available system documentation, they adjusted their anchors to come closer to acceptable solutions.
Impact of Traceability
Developers in the pure control group demonstrated relatively less appropriate adjustments compared to the treatment group. For example, developers in the treatment group who were anchored to the understanding that reservation is same as hold functionality adjusted this initial anchor after studying traceability knowledge, which revealed that hold functionality was already available. This led them to examine the change request and traceability knowledge thoroughly to understand the requirement accurately (adjusting flawed requirement anchor). Thus, despite their inappropriate initial anchors, these developers later adjusted to better reflect appropriate understanding of the task. This adjustment process was cumbersome for those who did not examine available trace-ability knowledge at all or used it as a last remedy. In most instances, developers in the pure control groups typically did not realize that their solutions were ineffective and incomplete, giving them a false sense of completion.
The extent of adjustment was primarily determined by the nature of traceability knowledge developers utilized in performing these tasks. Developers in the treatment group were able to adjust their anchors in most instances to more accurately reflect the current system state when compared to developers in the control and pure control groups. For example, for the reservation functionality, the high-end traceability model documents locations in the design that can accommodate the changes required whereas low-end practices simply provide links between the requirement for existing hold functionality and the design elements that implement this functionality. Specificity and the amount of knowledge provided by high-end traceability practices enabled developers to reach a more accurate solution as they were able to identify existing locations in design that can facilitate needed changes and leverage them in implementing the new functionality. Due to the effort required to study this information, some developers were reluctant to examine it until it became absolutely essential that they were unable to perform the task without consulting the traceability knowledge. But when they did use it, they usually realized that their anchors were not good enough and adjusted to arrive at more accurate anchors.
Evidence from our study demonstrates: 1. Both the presence and the interplay among anchoring and adjustment, availability, and confirmation biases in software development. 2. That traceability knowledge reduces the adverse effect of these biases on complex maintenance tasks. This was not the case for simple tasks where understanding the traceability knowledge demanded more cognitive effort than required by the task. Thus, for complex tasks, though considerable time is spent in understanding existing traceability knowledge, this results in higher quality changes when compared to quick-fix changes made without such knowledge.
Based on the findings from our study, we propose a causal framework that depicts the relationship between traceability, task complexity, cognitive biases, and maintenance performance (Figure 1). This model suggests that an improvement in traceability practices improves performance through the mitigation of cognitive biases and that this improvement is higher for complex tasks.
Drawing from our study, we suggest the following strategies to mitigate adverse effect of cognitive biases:
Software development organizations need to recognize the importance of the impact of developers' cognitive biases on the software development and maintenance process and take adequate measures to address them to improve performance. In this study, we have demonstrated traceability as an effective solution to address cognitive biases. This is one of the first studies that provides a theoretical motivation for the acquisition and use of traceability knowledge. The key implications drawn from this study highlight the need to establish and motivate developers to use tailored traceability practices. Future research should investigate appropriate incentive structures to motivate developers to follow such traceability practices.
8. Parsons, J. and Saunders, C. Cognitive Heuristics in Software Engineering: Applying and Extending Anchoring and Adjustment to Artifact Reuse. IEEE Transactions On Software Engineering, 30 (12), 2004. 873888.
a. In our study, individuals did not tend to overlook information that was inconsistent with self-generated anchors, rather they adjusted these self-generated anchors. These self-generated anchors can also be disrupted or changed by intervening variables.
©2008 ACM 0001-0782/08/0900 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.