Fairness in AI is harder than it looks

AI systems increasingly influence decisions in healthcare, hiring, lending, insurance, education, and criminal justice. Ensuring that these systems behave fairly across demographic groups is therefore both a technical and societal issue.

Unfortunately, “fairness” in machine learning is not a single well-defined concept. In practice, multiple fairness definitions exist, and under realistic conditions these can be incompatible with one another.

This article draws from the works of Barocas et al. (2020) and Corbett-Davies et al. (2023) to illustrate how a superficial take on algorithmic fairness can unintentionally harm the very groups it aims to protect.

A running example: diabetes screening

To ground the discussion in something concrete, let us borrow the diabetes example discussed by Corbett-Davies et al. (2023).

Suppose a hospital wants to determine whether a patient should be screened for diabetes. Screening requires time and medical resources, so unnecessary screenings should be avoided. At the same time, failing to identify diabetic patients can delay treatment and lead to serious health consequences.

The hospital wishes to follow the resonable policy:

Screen a patient if their risk is greater than 1.5%.

The data science team trains a machine learning model to predict the risk of diabetes for a patient given their attributes.

We will use the following notation:

• Y: the true outcome. Y=1: the patient is diabetic; Y=0: the patient is not diabetic

• R: the model's predicted risk of diabetes, ranging from 0% to 100%

• X: patient attributes (also called features or covariates), such as age, BMI, glucose levels, sex, race, ethnicity, and so on

• A: protected attributes - the subset of variables that may be associated with discrimination, such as race, ethnicity, sex, gender, religion, etc.

• P(⋅): probability notation.

For example: P( Y=1 | A=a ) means: “the probability that a patient is diabetic given that they belong to group a.”

In this running example, the protected attribute A concerns only race/ethnicity, with two groups: Asian-American and White-American. For brevity, we will simply refer to these groups as Asian and White, and to race/ethnicity as race.

Suppose epidemiological data shows that diabetes incidence differs between the groups:

P( Y=1 | A=Asian ) > P( Y=1 | A=White )

In other words, Asian patients have a higher baseline risk of diabetes ("base rate").

This difference in base rates will turn out to be central.

Independence, Separation, Sufficiency

In machine learning fairness literature, three major notions of fairness frequently appear.

Note that: Indepedence is also called "demographic parity"; Separation is also called "accuracy parity" in general terms (but can assume different forms depending on how it is exactly implemented, such as "equalized odds" or "equal opportunity"); Sufficiency is also called "group-wise calibration", as essentially it imposes that the machine learning model is calibrated across groups.

Important: Recall that calibration (the fact that we can interpreted R as representative of the real probability of the event Y) is not a given: a model may be accurate but not calibrated. Calibration is a central part of this article, review it on wikipedia (Öffnet in neuem Fenster) if you are unfamiliar with it.

We will now examine these criteria one by one, to assess their effect on the diabates example.

Independence (Demographic Parity)

Independence requires: R ⊥ A

This means the model’s predictions should be statistically independent of the protected attribute. In our diabetes example, this would mean that Asian and White patients are screened in equal proportions.

A common intuition is to implement independence through so-called fairness through unawareness: simply remove the protected attribute from the training data. If the model cannot see race, perhaps it cannot discriminate based on race. Unfortunately, there are problems in trying to achieve independence. And perhaps even more concerning, undesirable consequences may arise when independence is achieved. Let us review the first issue, then the latter.

Realizing independence

Realizing independence is non-trivial. There are two main notorious problems around it.

The first problem is the "proxy-attributes problem". Even if race is removed explicitly, other variables may still act as proxies for it. Examples include language, income, diet, geographic location, habits. A sufficiently powerful model can often reconstruct protected information indirectly. As a result, removing A from the inputs does not guarantee R ⊥ A. In practice, achieving demographic parity may require identifying and removing proxy variables as well.

The second problem is the "accuracy drop problem". Protected attributes may contain genuinely predictive information. For example, biological differences across populations may correlate with disease prevalence, treatment response, or symptom presentation. Counterintuitively, explicitly using protected attributes can sometimes improve fairness, because it allows the model to account for real differences in underlying risk distributions. Removing such information may reduce predictive performance and harm patients overall.

Consequences of imposing independence

Say independence can be successfuly implemented. What then?

Recall that the base rate of diabetes differs: P( Y=1 | A=Asian ) > P( Y=1 | A=White )

Independence (demographic parity) means both groups will be screened at equal rates. But the base rate of diabetes is different between Asians and Whites: the model thus ignores the fact that Asian patients are, on average, at higher risk. As a consequence, achieving independence may result in high-risk Asian patients not be screened, and low-risk White patients be screened unnecessarily.

This illustrates a recurring theme in AI fairness:

equal treatment does not necessarily imply equitable outcomes.

Separation (Accuracy Parity)

Separation requires: R ⊥ A | Y

This means that, conditional on the true outcome, predictions should behave similarly across groups. Intuitively: the model should make mistakes at similar rates across demographic groups.

This notion underlies metrics such as Equal Opportunity (equal true positive rates across groups) and Equalized Odds (equal true positive & negative rates across groups). In short, some measure of accuracy of the model should be equal across groups.

Realizing separation

In practice, separation can be realized with methods applied ante-training, such as group-aware loss reweighting, oversampling of minority group data (duplication or e.g. via synthetic data generation). Separation can also be achieved post-training, with methods such as Equalized Odds adjustments, that tailor group-specific thresholds for materializing the recommendation from R (e.g. "screen" at R > 1.3% for Asian, "screen" at R > 1.6% for White).

Consequences of imposing separation

Separation addresses an important concern: no group should disproportionately suffer from model mistakes.

In our diabetes example: we do not want Asian patients to be denied screening more often than White patients when they need it (are diabetic); nor do we want one group to be systematically over-screened when they do not need it (are not diabetic).

However, separation introduces another issue. Recall that the hospital's intend policy is to screen patients' whose risk exceeds 1.5%.

For this policy to make sense, the predicted risk must correctly represent the true risk P(Y) for any patient, be them Asian or White. However, this is precisely what sufficiency - and not seperation - guarantees: Y ⊥ A | R.

Unfortunately, it turns out that separation and sufficiency cannot be achieved together. More than that:

Independence, separation and sufficiency are pairwise incompatible when base rates differ

Let us sketch how this is the case between separation and sufficiency.

Separation is R ⊥ A | Y and sufficiency is Y ⊥ A | R. Having both means we're asking that the predicted risk does not depend on race when considering the true outcome, and that the true outcome does not depend on race when considering the predicted risk. It can be shown that for both to hold simultaneously, it must be Y ⊥ A: the base rate does not differ across groups. However, this is precisely the opposite of our situation: Asian and White patients are subject to a different incidence of diabetes.

Therefore when we enforce equal error rates, we change what R means across groups. This makes it hard to enforce the intended policy because, at least for one of the two groups, an R of 1.5% will not mean a true risk of 1.5%.

Sufficiency (Calibration Parity)

Sufficiency requires: Y ⊥ A | R - a prediction score should mean the same thing across groups. Sufficiency can therefore be viewed as a form of group-wise calibration. We have already discussed sufficiency in our comparison with separation just above, thus we will keep this section short.

Realizing sufficiency

Sufficiency is a group-wise version of calibration. Calibration is typically realized post-training, by applying methods like Platt scaling and Isotonic regression on top of the model. These produce a new R that is a P(Y)-aligned version of the "raw" R produced by the model. For sufficiency, group-aware variants of such calibration techniques must be set in place.

Consequences of imposing sufficiency

Sufficiency makes the model’s outputs interpretable and actionable across groups. In our diabetes example, sufficiency aligns naturally with the policy of screening a patient if their risk exceeds 1.5%, because R correctly represents P(Y) across both groups.

However, as established above, sufficiency comes at a cost: maintaining calibration generally leads to unequal error rates. In other wrods, imposing that R means the same for Asian and White patients means accepting that one of the two groups will experience more unecessary/missed screenings than the other.

Take-home: No silver bullet & the way forward

The incompatibility between independence (demographic parity), separation (accuracy parity), and sufficiency(calibration parity) under varying outcome prevalence across groups is a foundational impossibility in algorithmic fairness. This is a reality in many applications, as demographic groups often exhibit different outcomes due to biological, environmental, socioeconomic, historical, and structural factors.

Fairness is rarely as simple as "removing bias" from an algorithm, and fairness metrics should not be used as interchangeable levers to optimize mechanically. Instead, they should first serve as diagnostic tools, complemented by rigorous data and causal analyses. These analyses help us understand why disparities exist in the data and how models perpetuate or mitigate them.

The way forward entails gathering the most comprehensive picture possible: what disparities exist, why they arise, how they manifest, and what role AI plays in mitigating or exhacerbating them. This will ultimately allow us to can make reasoned, informed policy decisions about the trade-offs we’re willing to accept in the context at hand. Indeed, such policy decisions will need to seek equitable outcomes - and not subscribe to some statistically-sound but practically harmful treatment. Last but not least, monitoring should set in place to assess whether the policy will work and keep working as intended.