The second focus of my research is to study how technical definitions of fairness trade off against other properties of ML systems such as stability. The intuition is that we should only care about whether the outcomes of a decision-making system are ``equally good'' for all people (fair), if the outcomes are good to begin with. Put differently, a system that is completely arbitrary, i.e., always returns a random guess, is trivially fair, and so fairness considerations need to be complemented with other important measures of predictive performance.

Validity is one such important property. We interrogated the validity of two real-world personality prediction AI used in hiring (called Crystal and HumanticAI, and advertised to be used by several Fortune 500 companies) by looking at the stability of the outputs they produce. One (among several) shocking findings of our paper was that these systems produce arbitrarily different scores for resumes that are identical in every other manner except file type (pdf vs rich text)! This finding illustrates that if outcomes are arbitrary then it is meaningless to think about fairness.

Paper: Alene Kellogg Rhea, Kelsey Markey, Lauren D’Arinzo, Hilke Schellmann, Mona Sloane, Paul Squires, Falaah Arif Khan, and Julia Stoyanovich, ”An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring.” Data Mining and Knowledge Discovery, Special Issue on Bias and Fairness (2022)

Github: https://github.com/DataResponsibly/hiring-stability-audit

From a conceptual standpoint, I'm also interested in the converse question: how do fairness interventions affect stability, i.e., does imposing fairness requirements make outcomes more (or less) arbitrary for certain social groups than for others? One way to think about this is through the lens of the elegant bias-variance decomposition of model error. I ask the question of whether this decomposition is different for different subgroups in the data.

Working Papers: Falaah Arif Khan, Denys Herasymuk, and Julia Stoyanovich. 2023. On Fairness and Stability: Is Estimator Variance a Friend or a Foe? arXiv (Feb. 2023)

Falaah Arif Khan and Julia Stoyanovich. 2023. The Unbearable Weight of Massive Privilege: Revisiting Bias-Variance Trade-Offs in the Context of Fair Prediction. arXiv (Feb. 2023)

Github: https://github.com/DataResponsibly/Virny

The third line of my work focuses on how different data interventions in the model lifecycle affect the fairness and arbitrariness of predictions. This work takes a data-centric view, framing each stage in the lifecycle as a data transformation and studying its bias-transforming or variance-transforming effects.

It is a widely held socio-technical belief that data from socially marginalized groups tends to be more noisy, and this can manifest downstream as model unfairness. We interrogated this claim, and the impact of automating the detection and cleaning of data errors on model fairness. Counter-intuitively, we found that the incidence of automatically detected data errors --- missing values, outliers and mislabels --- does not track protected group membership, and yet the downstream impact of automated cleaning of these errors is more likely to worsen fairness than to improve it! We posit that this could be due to one or both of the following reasons: data errors for marginalized groups may be harder to detect or they may be harder to repair, impacting model fairness downstream. This result demonstrates the need to adopt a lifecycle view of fairness (e.g., to look at the interactions between error detection and data cleaning on downstream performance).

Paper: Shubha Guha, Falaah Arif Khan, Julia Stoyanovich, and Sebastian Schelter. 2023. "Automated Data Cleaning Can Hurt Fairness in Machine Learning-based Decision Making". IEEE ICDE’23

In this work we investigated the efficacy of a novel “who I am/how I behave” authentication paradigm. Conventional authentication works on a “what I know” (username/password) or “what I have” (device) model. Our system would study the user’s behavior while typing his/her username and use the activity profile as the key against which access was granted. This eliminated the need for the user to remember a password or have access to a registered device. Conversely, even if a password is cracked or a device is stolen, the bad actor would not be able to penetrate the system because his behavior would intrinsically differ from that of the genuine user.

Paper: Falaah Arif Khan, Sajin Kunhambu and Kalyan G. Chakravarthy. (2019) Behavioral Biometrics and Machine Learning to Secure Website Logins. SSCC'18

US Patent: Falaah Arif Khan, Sajin Kunhambu and Kalyan G. Chakravarthy. Behavioral Biometrics and Machine Learning to secure Website Logins. US Patent 16/257650, filed January 25, 2019

CAPTCHAs, short for Complete Automated Public Turing Tests to tell Computers and Humans Apart, have been around since 2003 as the simplest human-user identification test. They can be understood as Reverse Turing Tests because in solving a CAPTCHA challenge it is a human subject that is appearing to prove his/her human-ness to a computer program.

Over the years we have seen CAPTCHA challenges evolve from being a string of characters for the user to decipher, to be an image selection challenge, to being as simple as ticking a checkbox. As each new CAPTCHA scheme hits the market, it is inevitably followed with research on new techniques to break these challenges. Engineers must then go back to the drawing board and design a new and more secure CAPTCHA scheme, which, upon deployment and subsequent use, is again, inadvertently subject to adversarial scrutiny. This arduous cycle of designing, breaking and then redesigning to strengthen against subsequent breaking, has become the de-facto lifecycle of a secure CAPTCHA scheme. This beckons the question; Are our CAPTCHAs truly “Completely Automated”? Is the labor involved in designing each new secure scheme outweighed by the speed with which a suitable adversary can be designed? Is the fantasy of creating a truly automated reverse Turing test dead?

Reminding ourselves of why we count CAPTCHAs as such an essential tool in our security toolbox, we characterize CAPTCHAs in a robustness-user experience-feasibility trichotomy. With such a characterization, we introduce a novel framework that leverages Adversarial Learning and Human-in-the-Loop Bayesian Inference to design CAPTCHAs schemes that are truly automated. We apply our framework to character CAPTCHAs and show that it does in fact generate a scheme that steadily moves closer to our design objectives of maximizing robustness while maintaining user experience and minimizing allocated resources, without requiring manual redesigning.

US Patent: Falaah Arif Khan and Hari Surender Sharma. Framework to Design Completely Automated Reverse Turing Tests. US Patent 16/828520, filed March 24, 2020 and US Patent (Provisional) 62/979500, filed February 21, 2020