Directions of Technical Innovation for Regulatable AI Systems

What we know how to do.  We have proxies for checking many properties in these criteria (for example, data privacy, label quality, feature selection, fairness, and so on) using exploratory data analysis. As examples, we can inspect the annotation process and check inter-annotator agreement to get an idea of label quality.⁵ We can also measure (and correct for) imbalance in data if we are given group labels that segment the dataset. We have techniques for identifying influential points,⁶ outliers,⁷ and mislabeled points,¹¹ which may cause models to exhibit poor performance or bias. Further, there exist standards for reporting dataset information.⁸

Directions requiring additional AI innovation: 

• Metrics and generalizability. More work is needed to connect data metrics with impact on outcomes. For example, if a traffic-image dataset has a certain annotator disagreement score, what does that imply for an autonomous vehicle with a vision system trained on that data? Metrics along these lines are needed to capture which applications a dataset may be safely used for.

• Data quality checks in the context of pretrained models. Given the prevalence of pretrained models and the (currently) limited transparency about their training data, can we develop data checks that rely solely on accessing the model, or do we require disclosure of certain information about the training data? Do checks for fine-tuning data differ from checks for pretraining data?

• Unstructured data. For unstructured data such as images or social media messages, existing standards (such as those above) focus on reporting the statistics of the metadata. In our traffic-images example, is it sufficient to provide information about, for example, where the images were collected? Or should we also be providing information about the statistics of the image data itself?

Areas that require interdisciplinary engagement.  The specific metrics that would enable meaningful inference about the quality of the data will depend on the application. Questions around bias and fairness are also inherently multifaceted and use-case dependent. Determining how data collection respects copyright, obtains appropriate consent (opt-in vs. opt-out), and avoids misrepresentation of or detriment to the owner requires legal and social science input.¹⁵ Privacy tensions—what data is retained, what statistics are made public, what kind of access is granted to trusted auditors—must also be resolved within the broader sociotechnical context.

What we know how to do.  Given a specific metric, it is relatively easy to put the monitoring into place. Methods exist that establish distributions for “normal operation” and flag anomalous values during actual operation.²⁰ These techniques can be employed to detect shifts in top features, inputs, outputs, model confidences, calibrations, and fairness metrics. Knowing the causal structure of the environment, monitoring checks can specifically identify new confounders and mediators. In reinforcement learning (RL) settings, we can monitor whether reward distributions have changed. More generally, many best practices exist for testing AI systems.³⁸ For example, AI developers should test their systems with multiple external datasets, and results should be stratified by task difficulty and subpopulations of interest.

Directions requiring additional AI innovation: 

• Monitoring many metrics. Monitoring multiple metrics increases the risk of false positives, overwhelming engineers. How can we monitor many metrics efficiently while not incorrectly flagging too many cases for review? Relatedly, once in operation, what data should be gathered so that we can check additional metrics in the future? For example, consider an autonomous vehicle that has a very large number of machine learning (ML) components, processing data at fast rates: What should be stored? These questions remain despite advances in MLOps.

• Certification of use cases. How can we certify—or at least provide appropriate confidence for—use cases for which an AI system is supposed to work well, for example, where an autonomous vehicle should be able to drive safely? Certifying neural networks for safety-critical systems is an active research area,²¹ as are attempts to define appropriate levels of certification. Innovation to prevent model misuse is also needed.

• Correcting models after deployment. There exists some work on correcting deployed models in a way that does not require retraining end to end, including unlearning and fine-tuning.²⁷ But more work remains to be done, especially for AI systems with many interacting parts.

• Identifying relevant distribution shift. Many shifts are possible, and they may take many forms. Input distributions, relationship between inputs and outputs, and rewards (objective) may all change. For example, in newer cars, acceleration and popular colors may be different. Can we identify relevant and irrelevant shifts (for example, along the lines of Chuang et al.¹⁴)? If shifts occur in an uninterpretable embedding space, how can we explain them?

• Monitoring online learning agents. Beyond major adverse effects, how can we identify more subtle issues, such as initial signs of catastrophic forgetting, cheating, and other harms that occur while the agent continues to perform well on its reward metric? We need more work on identifying unintended consequences early.⁴¹

Areas that require interdisciplinary engagement.  There will always be a need to make high-level decisions regarding what should be monitored in a specific context and what safety assurances or guarantees are required, for example in healthcare.¹⁸ It is crucial to translate the monitored metrics into meaningful implications that enable people to make informed decisions within the broader sociotechnical system.

What we know how to do.  We can build inherently interpretable models (for example, generalized additive models, decision trees, rule-based models) for tabular and structured data.³⁴ We have some tools for interpreting neural networks in terms of human-understandable components, such as circuits⁴⁴ or even natural language.⁹ We can partially explain neural networks, for example, by visualizing weights or computing concept-activation vectors.³⁵

Directions requiring additional AI innovation: 

• Inherently interpretable models for more data types. Building interpretable models for non-tabular data (for example, images or audio) is still nascent. Learning concepts on top of the input dimensions may be useful.

• “Openboxing” large models. Can we build interactive, hierarchical, and semantically aligned views of large models, such that these models are to some extent inherently interpretable? For example, a traffic-image classifier recognizing objects by multiplying object templates with transformation matrices would be inherently more explainable than another model without such hierarchical structure. Can we allow users to explore explanations at different levels of fidelity for different contexts? Techniques exist but have limitations.²⁹

• Checking value alignment. Whether it is criminal justice, benefits allocations, or autonomous driving, AI systems are increasingly used in situations that require value judgments. How do we elicit and encode societal and individual values in diverse situations? What metrics can effectively measure value alignment? How do we make this mapping transparent so that others can understand the value choices made (for example, the drivers of other cars next to the autonomous vehicle)? Advancing existing work¹² is needed for our increasing use cases.

Areas that require interdisciplinary engagement.  There are the questions of what information to offer, and to whom. For example, releasing the code and environment may allow some to directly answer their questions.³² Providing an explanation broadens who can inspect the model, including users and domain experts; however, what information to release, how to extract it, and how often the information should be updated during the model’s life cycle will depend on the context. We will also need mechanisms for requesting more information about a model as new concerns emerge.

What we know how to do.  Given a distance notion, there are many techniques for providing local explanations. Specifically, we can find the closest point that leads to a desired output.²³ This is a counterfactual, and can help users determine what features set nearby points apart. It also lays the foundation for algorithmic recourse.²⁶

Directions requiring additional AI innovation: 

• Defining distance metrics. As noted earlier, local explanations rely heavily on notions of nearby data. It can be difficult to adjudicate what correlations in the data should be preserved and what should not. For example, if there are correlations between the kind of sign and the geographic location in a traffic-image dataset, should those correlations be retained in the distance metric? Some work exists on using human input to define the appropriate distance metric for the purposes of explanation and recourse,²⁶ but more work is needed.

• Data without interpretable dimensions. The challenges of choosing distance metrics become more complex when the data dimensions are not interpretable, such as pixels in the traffic images described earlier. What is a meaningful explanation in this case? Does it take the form of other images in the dataset? Should the input first be summarized into interpretable concepts?²² Similar issues arise with text and time series.

• Provenance adjudication. We may want to know if a particular input was used in a particular way to create a particular output (for example, in copyright). Beyond small models, this approach is in a very nascent stage.⁴³ 

• Trade-offs between explainability and privacy/security. Releasing information for audit or recourse may allow bad actors access to private information or to game the system.³⁷ For example, explanations in the form of training samples, like those of the traffic images, may allow actors to not only learn how to trick the autonomous vehicle but also learn about other elements of those images (that are not road signs). Advancing existing work⁴² is necessary to understand the resulting dynamics.

Areas that require interdisciplinary engagement.  The biggest question in these criteria is what makes an explanation “meaningful.”³⁹ This definition will depend on the context of the task: What is a meaningful explanation for a loan denial may not be the same for a medical error. In some contexts, a single recourse may be sufficient; in others, it may be appropriate to provide multiple options. Also, recourse generated from a local explanation may not always be the right way to assist a user unhappy with a decision. For example, suppose someone believes a voice-based COVID test is in error about their disease status. Rather than providing an explanation of the voice features used to make the decision, the appropriate recourse may be to allow that person to take a traditional COVID test instead.

What we know how to do.  In some cases, it is possible to disaggregate a complex task into simpler components. For example, we might evaluate an autonomous vehicle for its ability to identify and forecast the trajectories of other objects in its environment, and a planner’s ability to make safe decisions given this information. Algorithms for multi-objective optimization can find a Pareto front of options corresponding to different trade-offs between desiderata.³⁶ There is also recent work in inferring what objectives are truly desired given observed reward functions.³

Directions requiring additional AI innovation: 

• Metrics for metrics: Measuring match to goals. What are the measures that can be used to determine whether some technical objective matches our policy goals? Objective and reward design are relatively well-studied in some domains, such as RL,²⁴ but unsolved for many more situations—from autonomous vehicles to email text completion—in which we see AI systems used today. Further, our goals may be multifaceted; the objective must not only be faithful to our goal but also transparent in how it is faithful.

• Robustness to various objectives. Further research is necessary to create agents that excel across a range of objectives. In RL, this research can strengthen the robustness of learned policies when objectives are not perfectly specified.³⁰ It also applies to language models, which are trained to perform next token prediction but asked to perform tasks with various constraints and objectives and in many different worlds.¹ 

• Computational constraints for more robust objectives. Related to the above, are there computational constraints (for example, Lipschitz, sparsity, uncertainty) that might prevent the agent from overfitting to an imperfectly specified technical objective in ways that tend to align with broader policy goals?

• Understanding connections between objectives and learned model behavior. Can we efficiently explain how changes in the objective function affect model behavior? Conversely, can we explain policies in terms of compatible reward functions? Can we efficiently identify where two reward functions may result in different policies in human-understandable terms? Some prior works¹⁹ try to answer this; however, more analysis will help refine the reward function to better match the intended objectives. A further task is disentangling the influences of objective and training data on model behavior. For example, the mix of possibly conflicting beliefs in a text corpus will influence how language models trained on it behave, though all have the same objective.¹ 

• Inferring goals from observed behavior. We may have examples of decisions that we know align with the true goal (for example, safe driving behavior). However, the inverse problem of inferring rewards from behavior is not identifiable. Advances in inverse RL³ are needed to ensure that a learned reward aligns with the true goals.

Areas that require interdisciplinary engagement.  Designing objective functions involves evaluating their relevance, feasibility, and consistency with broader policy goals. Often, even policy goals can be vague, complicating the development and validation of AI systems. For example, in criminal justice applications of AI, the definitions and metrics for crime can be ambiguous, and the data used may not accurately represent the judges’ true goals.²⁵

What we know how to do.  Differential privacy¹⁷ is a widely accepted theoretical notion of privacy. In settings where this notion of privacy is appropriate, we have differentially private algorithms that can calculate statistical properties of data, train machine learning models, and generate synthetic data. Other privacy notions also exist; choosing which to use in a particular setting remains an open question.

Directions requiring additional AI innovation: 

• Better trade-offs between privacy and performance. In general, differentially private models have worse predictive performance than non-private models. How can that gap be closed? Can we ensure models remain private even with many queries and in conjunction with public data? What can we maximally expose about a model and training data? Can we precisely state what cannot be exposed, for example, an omitted long tail?

• Creating and assessing privacy definitions. How can we define privacy appropriately and meaningfully for different types of data, such as trajectories or text? What do current privacy definitions achieve on this data?

• Privacy via minimal data collection. Can we collect different information for different individuals to provide the AI system with the minimal necessary data required for making accurate predictions?

• Private generative models. Existing work focuses on classification. Questions remain when it comes to the privacy of large generative models: Can we prevent a generative model from replicating training data? Is there a difference between a private generative model and adding noise to data? Are empirical methods to ensure privacy, for example via reinforcement learning with human feedback, sufficient?

• Effective unlearning. In some cases, we may wish for people to elect to remove the influence of their data on the model after training. Unlearning methods are still nascent, especially for generative models.²⁷

Areas that require interdisciplinary engagement.  Current private models still allow third parties to infer private information via access to additional publicly available data. We need to develop new notions of privacy for this setting. Broader discussion is also needed around what to do if privacy guarantees sacrifice predictive performance, especially if the sacrifice primarily affects underrepresented groups.⁴ More generally, the appropriate definition of privacy, and how strict the privacy guarantee must be (for example, via hyperparameter settings), will depend on the setting.

What we know how to do.  There has been significant work on learning from humans. We can apply methods such as imitation learning and reinforcement learning from human feedback²⁸ to orient the model based on expert control or learn human intentions. Active learning techniques can be used to proactively ask for information from humans to improve a model.³³ We also have methods for humans to take the initiative to correct an agent. While methods in uncertainty quantification are always being improved, for the purposes of flagging uncertain inputs for human inspection, our current methods are reasonable.

Directions requiring additional AI innovation: 

• HCI methods for avoiding cognitive biases. Humans have many cognitive biases and limitations. If a system behaves most of the time, people may start to over-rely on it. Confirmation bias can accompany backward reasoning (people finding ways to justify a given decision) but can be mitigated by forward reasoning (looking at the evidence).¹⁰ Bias can also come from imperfect information fusion; for example, if a human inspects the input data and then views an AI prediction based on the same input data, they may falsely believe that the AI prediction is a new, independent piece of information. Appropriate human+AI interaction can help mitigate these biases.

• Semantic alignment and shared mental models. We need shared mental models among the human, the agent, and the methods to ground terms.² Some works use models of humans to facilitate interaction, including modeling a person’s latent states such as cognitive workload and emotions.³¹ But it remains an open question as to how to develop and validate these methods for increasing numbers of human+AI use cases.

• Time-constrained settings. How can we design safe and effective hand-offs between AI systems and humans in time-constrained settings, such as AI-assisted driving in emergencies?

• Test-time validation of large surface models. Models with large output surfaces (for example, LLMs) will be difficult to evaluate via prospective metrics; we need methods to assist people in their validation at task time.¹⁶ 

• Evaluation and design of realistic human-in-the-loop systems. Testing various forms of shared control (for example, an AI-assisted driving system) requires significant training and risk. How can we make it more efficient?

Areas that require interdisciplinary engagement.  Human+AI decision making is highly interdisciplinary, involving social and cognitive science, psychology, and other related fields. Fortunately, HCI research has existing connections to these fields.⁴⁰ Furthermore, adoption of new tools into workplaces is well studied in design, human factors research, management, and operations science. Whether, how, and which humans to include in the loop, as well as how AI systems should respond to inappropriate, slow, or absent human input will require interdisciplinary efforts.