Teaching the FATE Community about Privacy

https://bit.ly/3IWHvbW April 13, 2023

I have worked on privacy for approximately 20 years. I have recently started looking at issues in Algorithmic Fairness, Accountability, Transparency, and Ethics (FATE), and I believe there are many lessons that the FATE community can learn from past work, successes, and failures in privacy.

Back to Top

How Are FATE and Privacy Similar?

FATE and privacy are problems that organizations are struggling with, with failures ending up as headline news. Both are also ill-defined. There are many specific and narrow definitions of privacy, for example anonymity, the right to be forgotten, and contextual integrity, but no single widely accepted definition. The same is true of FATE. There are many useful concepts, such as allocation harms vs. representation harms and equalized odds, but no single framing that covers the wide range of issues.

FATE and privacy are also non-functional requirements that cut across the entire software development lifecycle. Functional requirements are features that are often in a specific part of the source code and have a clear criteria of whether they work or not. In contrast, non-functional requirements are cross-cutting and sometimes emergent properties of a system, span many parts of the system, involve many sections of source code, are hard to measure, and are not necessarily anyone’s specific responsibility.

Lastly, developers lack awareness and knowledge of FATE and privacy issues. Many studies have found that developers have little knowledge of privacy (for example, see Li, Agarwal, and Hong⁴). Informally, we have seen the same with FATE. In interviews with Machine Learning (ML) developers to understand how they evaluate their systems,² very few raised issues of fairness or bias. In both cases, while there are many aspirational guidelines (think of Asimov’s Three Laws of Robotics), there is also a lack of practical guidelines and best practices for developers.

Back to Top

What Did and Did Not Work for Privacy?

There has been much research probing people’s privacy concerns, and proposals for new user interfaces, data-processing techniques, and system architectures. Most of this research was good in raising awareness in general and for regulators specifically, but generally did not have much success in moving the needle.

Privacy policies have been widely adopted and are now commonplace. This kind of transparency is good in theory, but less so in practice since it places the onus of privacy on end users. In general, attempts to improve privacy by helping end users have not worked, since most people don’t have the time, expertise, or desire to deal with all the nuances of privacy.

Industry has also tried to self-regulate, for example with Web standards like P3P and Do Not Track. However, these pushes repeatedly failed due to misalignment between the various stakeholders and consumers. In particular, advertisers have an extremely strong incentive to collect as much sensitive data as possible. More importantly, these standards had no enforcement mechanisms or penalties for lack of compliance.

So what has actually worked for privacy? Market forces have had a small positive impact on privacy, but it is still difficult to compete on privacy. Imagine you want to buy a webcam. You can easily compare price, form factor, and color, but it’s near impossible to compare privacy. Apple’s Privacy Labels and Google’s Data Safety Section for smartphone apps, as well as privacy nutrition labels for smart devices,³ are attempts to ameliorate this problem. I believe these labels will be effective in influencing product managers who will want to compete on privacy and help product reviewers incorporate privacy into their reviews.

Social forces have also had a small positive effect on privacy. Perhaps the most effective tactic has been shame. For example, my team built PrivacyGrade.org to grade the privacy of a million Android apps. Our intent was to raise people’s awareness of privacy, but one unexpected result was that some app developers publicly declared changes in their data collection behaviors, in part due to bad press resulting from our site.

Overall, the most substantive lever for improving privacy has been legislation such as GDPR and CalOPPA, along with regulatory fines by organizations like the Federal Trade Commission. A decade ago, a question I often got was how to get organizations and developers to care about privacy. Nowadays, companies and developers must care about privacy, due to the potential for massive fines. While I am skeptical of some parts of GDPR and CalOPPA (do those cookie notices help anyone?), they have forced companies to think more about what data they are collecting and why. It’s worth noting this legislation and regulation happened because of repeated data breaches and privacy abuses, as well as industry’s inability to self-regulate.

The second-most-effective lever for privacy has been smartphone app stores. The centralized nature of app stores and their commanding position in distributing apps made it possible for Apple and Google to dictate standards for privacy. Developers that did not comply would simply have their app removed from the app store.

Back to Top

What Might Work for FATE?

So, what can we learn from privacy that might work for FATE? One is that industry self-regulation is unlikely to work. While there is strong desire from industry to improve, FATE is too diverse, with too many actors across too many kinds of systems and industry sectors. Industry simply has no way to monitor the sheer number of algorithmic systems being deployed or sanction actors that fail to comply. There also is not a centralized app store or platform where a small number of organizations can dictate and enforce policy.

Market forces also are unlikely to make a large difference. There are no clear metrics to compare fairness or ethics of different algorithmic systems. Furthermore, the sheer diversity of algorithmic systems coupled with a lack of alternatives in many domains (for example, algorithms for health care or housing for the homeless) makes it hard for the market to solve FATE issues.

Social forces have been surprisingly effective in getting companies to respond to FATE issues. In particular, audits by third parties have had success in getting companies to change, and it’s worth investigating how to support these audits more. Some of these audits have been conducted by experts. For example, independent researchers found that many commercial face-recognition systems did poorly on people with darker skin and with women.³ This work led Microsoft, Amazon, IBM, and others to revise their face-recognition systems or halt their services. Other audits have been conducted by everyday users. My colleagues and I studied how people on Twitter organically came together to audit things like Twitter’s photo cropping algorithm and Apple’s credit card algorithm.¹ These audits were surprisingly effective, in that they led to popular press and interventions either by the company involved or by government agencies.

In the long term, legislation and regulation will be key to making substantive improvements in FATE, mostly because the other knobs and levers will be insufficient. If FATE follows a similar trajectory as privacy, there will be many more years of embarrassing headline news of FATE failures and a continued inability by industry to self-regulate before policy makers finally step in.

While I may have seemed negative on the value of privacy research, it’s more that research by itself is usually not sufficient. One way research can help is to be more immediately actionable. For example, how can the research give more ammunition for policymakers? How can a new tool be designed to streamline adoption?

Lastly, there is potential synergy for FATE people to work closely with privacy advocates already in an organization. Both are interested in transparency, minimizing risks to users, and proper use of sensitive data. This approach may make it easier for FATE to be addressed in an organization, as privacy engineers and chief privacy officers are starting to become commonplace and much of their work is focused on compliance with policies and procedures.