Are Software Developers Affected by Cognitive biases while performing changes to design artifacts? If so, how can we mitigate the impact of such biases on developers’ performance? Despite the criticality of these questions, they have been relatively unexplored in the context of software development. When making changes to software artifacts, developers use heuristics that are influenced by cognitive biases. The goal of this article is to examine this influence and to explore whether such biases can be mitigated by using cognitive aids such as traceability (See accompanying sidebar). This article describes the findings of our study and provides recommendations for practice.
Cognitive Biases in Problem-Solving
People employ cognitive biases to reduce uncertainty and complexity when processing information during problem solving.11 While the effects of these biases have been well studied in general problem-solving, their impact on software development has gained limited attention. These biases are often employed to simplify complex inference tasks to more manageable proportions.8 The most common cognitive biases experienced during problem-solving include anchoring and adjustment, availability, and confirmation.
This bias results from people forming initial estimates about a problem and adjusting their initial estimates to arrive at more appropriate final solutions.8,10 This initial estimate is called as an anchor – ‘a familiar/known position or a reference point’, that is subject to adjustments.5,7 Typically, these adjustments are insufficient.11 Anchoring and adjustment have been known to impact tasks like artifact reuse and decision making.8 Anchors can be self-generated or externally-generated. To form self-generated anchors, developers retrieve their long-held beliefs (say, past development and domain experiences) from their memory to draw inferences about a task. This is similar to the concept of conservatism. Conservatism is defined based on the following three aspects:
- The tendency of people to overlook information that is inconsistent with their long-held beliefs.4 People fail to revise their previous decisions, findings, or interpretations in light of new information and hold onto their longer-term opinions,1
- The time interval between the initial estimate and the judgment task is long,7 and a In our study, individuals did not tend to overlook information that was inconsistent with self-generated anchors, rather they adjusted these self-generated anchors. These self-generated anchors can also be disrupted or changed by intervening variables.
- Influence of this tendency on judgments over a longer period of time without being disrupted by intervening values.4,6
Self-generated anchors exhibit only the second characteristic of conservatism (the long time interval). The other two aspects of conservatism are not necessarily manifested by self-generated anchors.a This discussion implies that conservatism and anchoring are analogous but distinct concepts.
Externally-generated anchors, on the other hand, are triggered by certain artifacts such as system documentation, requirements specification, etc. used by developers. Developers may combine self- and externally-generated anchors or use externally-generated anchors to adjust self-generated anchors to arrive at acceptable solutions. Irrespective of the source, these anchors may be inaccurate and developers may be unaware of these inaccuracies and thereby not realize the need for adjustment. Specific mechanisms are needed to make developers aware of the same. Insufficient adjustments are made when there is an uncertainty about the correct solution. This may lead to premature termination of the task due to reluctance to spend more effort to adjust sufficiently or due to lack of appropriate cognitive resources to facilitate the process of adjustment.
This bias suggests that people are influenced by information that is easier to recall and are prejudiced by information that is well-publicized or recent.10 This suggests that in the absence of any usable knowledge about a system, developers may draw heavily upon their conveniently accessible past experiences. Such experiences may not always be consistent with the current system state and might have undesirable consequences. For example, it might result in erroneous changes to the system if their experiential beliefs are inconsistent with current system state.
This bias suggests that people tend to focus on information that is consistent with their preconceived notions while they disregard information that is inconsistent with their past conceptualizations.3 In other words, people mostly seek evidence that qualifies their beliefs rather than evidence that disqualifies them.1 Developers’ past beliefs might dictate the importance placed on new information about requirements, design, and solution.
Since the evidence of the impact of these biases in software development is largely anecdotal, we empirically investigated this phenomenon and the role of traceability in shaping these biases.
How Do Cognitive Biases Affect Software Development?
Findings from our study (see accompanying sidebar) demonstrate the presence of cognitive biases in software development. Anchoring and adjustment was found to occur at various stages during the process of making changes to system design. Developers alternated between anchoring and adjustment for different purposes (for example, to develop an initial understanding of requirements, system design, and task). Confirmation and availability biases were found to affect the process of anchoring and adjustment through the nature of knowledge available to and sought by the developers.
During each step in the process of handling a change, developers anchored to their initial understanding of the requirement (requirement anchor), current design artifacts (design anchor), and the solution for the change requested (solution anchor). For example, after developers read new requirement specification, they developed an initial understanding (requirement anchor) about the requirement even before actually examining system documentation to understand it in the context of the system. This is typically a combination of self and externally-generated anchors: self-generated from their preconceived notions about the domain and externally-generated by the implied semantics in new requirement specification.2 b
Impact of Traceability
Based on the verbal protocol analysis, we found that developers in the pure control group typically formed anchors that were different and often less accurate when compared to those in the treatment group. Let us consider one of the four maintenance tasks performed by the pure control group: “incorporation of reservation functionality in the library management system.” They formed an initial anchor that ‘reservation functionality’ is same as ‘hold functionality’ (requirement anchor). This was primarily triggered due to the disuse of traceabil-ity knowledge that could have helped them make the distinction between their belief and the actual functionality. Their interpretation was based solely on their level of familiarity and their mental model of library domain. They had no mechanisms to confirm or disconfirm their beliefs leading them to confide in their incomplete and inconsistent knowledge. They anchored to these estimates and made ill-informed changes that were inaccurate, incomplete, and inconsistent with the current system state. Although subjects in the control and pure control groups did not arrive at incorrect interpretations in every instance (say, when developers needed to understand design of the existing system and the impact of changes), they did so in most instances.
Developers in the treatment group formed more appropriate understanding of current design artifacts (design anchor) and the impact of the changes (solution anchor) more often than the control and pure control groups. For example, they used traceability knowledge to identify the impact of the introduction of the new functionality. Most often, they were also able to correctly identify alternatives available in current design that can be leveraged in implementing the new requirement as a result of the links provided by the traceability knowledge from related requirements to such alternatives. Developers in the control and pure control groups were usually unable to completely identify the impact of the ‘reservation functionality’ and made decisions based on their initial anchor of what needs to be changed. This observation of differences based on developers’ verbal protocols was consistent in most of the instances when subjects needed to understand existing system and the impact of changes. This behavior of developers corroborates the existence of availability and confirmation biases when anchoring to estimates of a solution and the role of traceability in mitigating the effect of these biases.
Developers iteratively adjusted their anchors as they navigated through various design artifacts. After examining a few design artifacts and without expending significant cognitive effort in understanding what might be relevant to the change request, developers initially formed anchors about the current system state. However, developers did not always adhere to these anchors. As they discovered new information from the design artifacts and available system documentation, they adjusted their anchors to come closer to acceptable solutions.
Impact of Traceability
Developers in the pure control group demonstrated relatively less appropriate adjustments compared to the treatment group. For example, developers in the treatment group who were anchored to the understanding that reservation is same as hold functionality adjusted this initial anchor after studying traceability knowledge, which revealed that hold functionality was already available. This led them to examine the change request and traceability knowledge thoroughly to understand the requirement accurately (adjusting flawed requirement anchor). Thus, despite their inappropriate initial anchors, these developers later adjusted to better reflect appropriate understanding of the task. This adjustment process was cumbersome for those who did not examine available trace-ability knowledge at all or used it as a last remedy. In most instances, developers in the pure control groups typically did not realize that their solutions were ineffective and incomplete, giving them a false sense of completion.
The extent of adjustment was primarily determined by the nature of traceability knowledge developers utilized in performing these tasks. Developers in the treatment group were able to adjust their anchors in most instances to more accurately reflect the current system state when compared to developers in the control and pure control groups. For example, for the reservation functionality, the high-end traceability model documents locations in the design that can accommodate the changes required whereas low-end practices simply provide links between the requirement for existing hold functionality and the design elements that implement this functionality. Specificity and the amount of knowledge provided by high-end traceability practices enabled developers to reach a more accurate solution as they were able to identify existing locations in design that can facilitate needed changes and leverage them in implementing the new functionality. Due to the effort required to study this information, some developers were reluctant to examine it until it became absolutely essential that they were unable to perform the task without consulting the traceability knowledge. But when they did use it, they usually realized that their anchors were not good enough and adjusted to arrive at more accurate anchors.
Summary of findings
Evidence from our study demonstrates: 1. Both the presence and the interplay among anchoring and adjustment, availability, and confirmation biases in software development. 2. That traceability knowledge reduces the adverse effect of these biases on complex maintenance tasks. This was not the case for simple tasks where understanding the traceability knowledge demanded more cognitive effort than required by the task. Thus, for complex tasks, though considerable time is spent in understanding existing traceability knowledge, this results in higher quality changes when compared to quick-fix changes made without such knowledge.
Based on the findings from our study, we propose a causal framework that depicts the relationship between traceability, task complexity, cognitive biases, and maintenance performance (Figure 1). This model suggests that an improvement in traceability practices improves performance through the mitigation of cognitive biases and that this improvement is higher for complex tasks.
Mitigating Cognitive Biases
Drawing from our study, we suggest the following strategies to mitigate adverse effect of cognitive biases:
- Provide tailored traceability: In our study, high-end traceability users adjusted most appropriately, producing highest quality changes. This suggests that development of high-end trace-ability models tailored to project-specific needs (say, product family development, agile development, etc.) becomes essential in mitigating cognitive biases. This reiterates the need to develop project-specific traceability environments.2
- Establish tight integration between design artifacts and traceability knowledge: One reason why developers are inhibited from using documentation is the weak connection between design artifacts and the traceability knowledge. Although traceability knowledge provided to the subjects in our study had significantly closer relationship to design artifacts than what is typical in the industry, it was apparent that tighter integration would have been much more valuable. If developers can navigate seamlessly between traceability knowledge and the artifacts that need changes, effort exerted for navigation will be lower, making traceability knowledge more accessible and available. This will help developers better comprehend the impact of their changes, thereby instigating them to adjust sufficiently.
- Provide visually supported traceability knowledge: It is essential to provide appropriate visual aids to depict impact of changes. Developers spent a major portion of time in finding answers to questions of descriptive nature (what needs to be changed and where artifacts to be changed are located) and of evaluative nature (evaluating the completeness and accuracy of tasks). By providing visually supported traceability knowledge, time expended to study traceability knowledge can be minimized.
- Motivate developers to use trace-ability knowledge: Typically, developers consider sifting through traceability knowledge as wastage of valuable time that could be spent in actually performing the modifications. However when handling complex maintenance tasks without traceability, developers spend significant amount of time in futile attempts to solve the problem only to later realize the usefulness of trace-ability in helping them identify the relevant artifacts to be changed. System quality is adversely affected by such ill-informed changes. This could result in further effort spent in future in refactoring the system. Traceability tool developers should develop environments that will induce developers to navigate through traceability knowledge before performing changes.
Conclusions
Software development organizations need to recognize the importance of the impact of developers’ cognitive biases on the software development and maintenance process and take adequate measures to address them to improve performance. In this study, we have demonstrated traceability as an effective solution to address cognitive biases. This is one of the first studies that provides a theoretical motivation for the acquisition and use of traceability knowledge. The key implications drawn from this study highlight the need to establish and motivate developers to use tailored traceability practices. Future research should investigate appropriate incentive structures to motivate developers to follow such traceability practices.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment