The Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyond

The problem of vigilance.  As the degree of automation increases, humans increasingly take on the task of vigilance, becoming monitors of what the system is doing rather than active participants in the workflow. We know from very early vigilance studies that humans are very poor at these kinds of tasks. For example, a classic study¹⁴ found that when people were asked to monitor visual processes for rare events, even highly motivated subjects found it difficult to maintain attention for more than about half an hour. Further, research (for example, Warm et al.²⁵) shows that jobs that have a heavy monitoring requirement, with little in the way of direct engagement in activity, lead to high levels of stress and poor health, something that partly explains why jobs such as air traffic control are so difficult.¹³ When autopilot was first introduced in aircraft, concerns that this would turn humans into system monitors even led some HFE specialists to propose that, in the flight deck, we should ask whether it should be computers monitoring the pilots rather than the other way round.²⁶

Turning to today’s AI systems, we should similarly ask what AI applications mean for a world in which many of our jobs might increasingly require us to monitor or oversee what our intelligent systems are doing, with concomitant concerns about whether we will notice when we need to intervene and, more long term, whether our jobs will become more stressful and less satisfying as a result.

Partly as a result of the COVID pandemic, we are much more attuned now as a society to the importance of well-being at work not just for personal reasons, but also for the health and productivity of organizations. As Csikszentmihalyi⁵ noted in his studies, we are at our happiest when we are absorbed in tasks that not only allow us to exercise our talents but to stretch them. This calls for careful attention to how to design the human-machine partnership around the primacy of people, to ensure humans are engaged, that they can do what they enjoy and are good at, and that they can learn and grow as a consequence.

The takeover challenge.  Another major concern is attending to the hand-off between human and machine during the course of work. When trouble arises, humans often need to intervene, and when processes are heavily automated there are consequences for this transition when the human has been largely out of the loop. As Bainbridge highlighted,³ having to re-engage with an automatic system can be problematic because human operators have been paying attention elsewhere, meaning they have reduced situational awareness of what the requirements of the task are and what the context of the work is, undermining the human’s ability to take appropriate action.

In aviation, there have been notorious examples where pilots have not understood, paid attention to, or been properly alerted to the actions of the autopilot system. Then, when things start to spiral out of control (sometimes quite literally) the pilot is lacking in the necessary situational awareness and knowledge to step in and take the right corrective actions. A salient example is the case of China Airlines Flight 006 in 1985 where reliance on the 747’s autopilot during an engine failure introduced complexities contributing to disorientation of the pilot, leading to a sudden uncontrolled 30,000-foot plunge of the aircraft toward the ocean.

The influence of automation on human awareness and vigilance can be seen in other domains too, including our role as the operators of heavy equipment in our daily lives. For example, studies show that new forms of automation in our cars, such as adaptive cruise control and smart lane following, can have detrimental effects. For example, automobile drivers can become complacent and less aware of hazards with the use of adaptive cruise control while driving, resulting in a negative impact on safety, such as longer response times to hazards.^21,27

Extrapolating to AI systems more generally highlights the possibility that too much automation may create situations in which we over rely on systems to do our work for us, with the result that we have no clear understanding of the flow of work, where and how we should intervene, and a lack of resources in the moment to enable us to effectively respond, guide, or contribute. Keeping the human in the loop is therefore crucial not just for these reasons, but also because disengagement from tasks, where a person can otherwise exercise their skills, can lead to deterioration in well-being, mood, and creativity.⁶ Further, this points to the need for better capabilities on the part of the AI system for mutual grounding, feedback, and alerting.

De-skilling through automation.  Longer term, the ability to take over control is of course made worse if the flight crew has suffered from the deterioration of skills due to an ongoing lack of engagement in the flow of work. Bainbridge drew attention to the irony of not only physical skills deteriorating, but also cognitive skills waning when processes are increasingly automated. However, when automation fails, it is just these skills that are crucial when the human needs to step in and take control. The aviation industry has known this for decades,²⁶ hence the need for simulator training where faults are routinely injected into the system to ensure that pilots not only know how to manually fly a plane, but that they also know how to diagnose a problem. Overreliance on aviation autopilots is known to have serious downsides both in terms of pilots failing to develop an implicit understanding of how a system works, as well as undermining the problem-solving skills needed to critically evaluate the output of the autopilot and the state of the aircraft in order to take corrective action.

Extrapolating to human-AI systems underscores the fact that we understand very little about how the offloading of different components of our workflows to these new AI systems will impact our cognitive skills for a given task or job of work. The complexity and impenetrability of today’s AI systems already compromises their intelligibility for us, but if we consider that we might be less and less “hands on” as these technologies are harnessed to do more, we might fail to build effective, implicit understandings of them. Such understandings are built through our ongoing interactions with systems, helping us to learn how our actions relate to the output of the machine. And there are many other nagging, related questions too: How deeply will we understand or remember aspects of the domain we are working within if an AI system does the work? This may not matter for some kinds of tasks, but for others, we may only ever develop a shallow understanding of the subject of our work. Further, what critical skills will wither away and which new ones will we need to develop? There is little doubt that AI will shape our cognition, which in turn has important implications for education, skills training, and jobs of the future.

Trust and automation bias.  A fourth issue concerns the predisposition for the human user or operator to critically assess or question the output of the automated system. When autopilot was first introduced into aircraft, accident analyses indicated a key contributing factor was that pilots were overly reliant and overtrusting of the autopilot system. The pilots either blindly accepted its output, or failed to act unless the autopilot system advised them or alerted them to act—a phenomenon now known as “automation bias.” This perception of the superiority and correctness of the machine is one lesson from aviation. Another lesson points to the perception of the superiority of the pilot in relation to more junior crew being a contributing factor in catastrophic accidents. Lessons were learned some decades ago²² of the risks of junior crew not speaking up or questioning the actions of the captain. Perhaps the most notorious case is that of the Tenerife airport disaster in 1977, where the reluctance of junior crew to challenge the captain of the KLM aircraft was cited as a contributing factor to the deadliest accident in aviation history. Too much trust or deference, combined with the failure to question actions in the cockpit, whether by pilot or autopilot, can lead to critical incidents.

The analogy to AI systems here should be clear. We know already that people are prone to automation bias when it comes to working with AI, tending to favor and failing to question the output of the AI system (for example, Alon-Barkat and Busuioc¹). But there is much to learn about automation bias in human-AI workflows. For example, the timing of when AI output or recommendations are delivered—whether prior to human evaluation or as subsequent critiques—can impact human decision making in complex ways.⁸ Users are unlikely to recognize how such sequencing affects their judgments, and research is only beginning to uncover the underlying dynamics. More studies are therefore needed in order to design systems to engender the appropriate levels of trust in the output of the machine, and to support people in critically assessing the actions and recommendations of their AI partner.

Keep humans in the loop.  As we have described, existing research highlights the need for human engagement (whether mental or physical) rather than complete relegation to the role of overseer or monitor. As technologists become more ambitious about the degree to which AI systems can automate human activity, humans may, at the same time, start to monitor these systems less as they trust them more. But for all the reasons cited above, users need to be kept in the loop to both be and feel empowered, to implicitly learn about the system through interaction, and for well-being. More specifically:

• Be cautious about the amount of automation possible before humans are invited or required to intervene. For example, we know that continuous and ongoing interaction with an AI system can effectively and implicitly build users’ mental models of the system.²⁴

• Consider with care when it makes more sense that AI systems monitor humans, rather than the other way round. (This is after all what spell and grammar checkers do today, supporting and monitoring human action and offering assistance in an unintrusive way.)

Uphold the primacy of human agency and role allocation.  It is not enough that humans are engaged—we also must design a safe and effective partnership. Studies in HCI and HFE reach a similar conclusion on this point, but changes in the capabilities, pervasiveness, and agency of AI makes action in accordance with the findings and implications more urgent. First and foremost, as in aviation, the co-pilot must cede ultimate control and final responsibility to the human user as “pilot,” with the co-pilot acting in a well-defined supporting role. Designing the role of the machine also must take into account what people are good at versus the capabilities of machines, as well as consideration of what motivates and makes people happy. This means we must do the following:

• Design AI features and overall software applications they are hosted within to enable and celebrate the primacy of human agency. Achieving this goal can be challenging and thus may tend to be lost to simpler, easier integrations of AI into applications.

• Reinforce a clear hierarchy of control and decision making with the human in charge. Note that this does not mean that such oversight necessarily leads to better outcomes in all situations. In fact, research shows that predictive or diagnostic tasks may be more accurately achieved by the machine.¹⁰ However, such reliance on human oversight makes explicit that the pilot or human is the one held accountable, with the responsibility to ensure the best sources of information are evaluated and that the implications of output or recommended actions amid the complexities of the open world are considered.

• Allow users control over how and when they engage in the workflow and degree of oversight. For example, AI automation without detailed human oversight may be preferred for tasks and scenarios where AI behavior and goals are well-understood, stakes are low, and recommendations are guided via observation of user actions or by higher-level user controls.¹⁹

• Support users in carrying out aspects of tasks that allow for self-expression, creativity, social judgment, and navigating complex situations that extend beyond AI capabilities.

Building human skills and capabilities.  AI systems are now capable of taking on more of our human tasks than ever before, changing the skills we learn and maintain. However, when designed correctly, AI systems can enable us to build new skills rather than to undermine our existing capabilities. For example, educators are now exploring ways in which AI can be used as personalized coaches or tutors to boost our capabilities (for example, Mollick¹⁷). Consider the following:

• Though AI systems are developing sophisticated conversational capabilities, users nonetheless need to learn how to “speak machine” to best effect both in terms of learning how to most effectively prompt these systems as well as how to assess their actions.

• AI systems need to be endowed with the ability to collaborate with users to achieve mutual understanding about the intentions, goals and capabilities of users. One approach is the development of teachable AI systems (training the co-pilot) so that they are personalised and work for us in different contexts.¹⁸

• Consider the coaching metaphor—design systems that help us develop new skills that continue to exist even when not using the AI system.¹¹

Designing more intelligible systems.  By working to boost the intelligibility of AI systems, we support better situational awareness and mental models for users so that they can intervene when necessary, critically assess the output of the machine, and address problems when they occur. Tried and tested approaches from HCI are useful here, such as:

• Refer to well-established HCI methods and design guidelines, including principles specifically for mixed-initiative interfaces, a term coined by Horvitz¹² referring to systems where people and machines work in partnership toward shared goals. More recently, the set of principles for guiding designs for human-AI interaction were further extended and studied.²

• General HCI principles include making clear what a system is capable of, acknowledging user actions, providing effective and ongoing feedback to users, and delivering well designed explanations for system responses, and alerting users to problems or issues in intelligent ways, at the right time and with guidance as to how to intervene.

• While “explainable AI” is a burgeoning field, it is also clear that social science has a great deal to contribute here (for example, Miller¹⁶) and that we need to consider how to design the whole user experience, not just the output of the algorithms.²⁴

Designing for appropriate levels of trust.  The sheer complexity of today’s AI models, the fact that they are probabilistic, and their tendency to produce plausible but sometimes false output, calls for a renewed emphasis on designing for trust. We need to design systems and train users so that they develop appropriate levels of trust with a system, and so that we can avoid well-known biases such as automation bias. Users need to be helped to develop their critical thinking skills and not fall into complacency. Intelligibility in general helps here, but more specific design considerations, mostly drawn from Horvitz¹² and from Passi and Vorvoreanu²⁰, include:

• Take account of users who may vary in expertise, AI literacy, and task familiarity in designing the user interface and feedback from the system.

• Identify ways to assess automation bias from telemetry and monitor with changes to the interface and workflows. Use onboarding techniques and tutorials to make users aware that overreliance is a common phenomenon, giving them examples of correct and incorrect output. It is especially crucial to set the appropriate levels of trust in early use of a system. Be transparent about a system’s strengths and limitations, as well as intended uses.

• Endow systems with the capability to infer well-calibrated confidences in their recommendations and generations. Harness these confidences to guide, gate, and annotate output, including the sharing of indications of uncertainty with users.

• Findings on automation bias highlights the importance of investing in additional research, such as study of the value of computing and relaying well-calibrated confidences to users. We also need better understandings of ideal designs for human-AI workflows, including how different strategies for sequencing and presenting AI recommendations can support, distort, or hinder human judgments.