Fairness and Subgroup Evaluation

#Why averages are not enough

An overall performance score can hide important differences between groups. A model may miss more cases in one population or generate more false alarms in another. Relevant groups depend on the task and setting, and may include age, disability, sex, language, ethnicity or combinations of characteristics.

Unequal performance can arise from limited representation, different measurement quality or labels that reflect unequal access to care. Simply including more records does not necessarily fix these problems. Data may reproduce past inequalities even when group identity is not explicitly supplied to the model, because other inputs can act as indirect signals.

#What subgroup checks can show

Subgroup evaluation can compare sensitivity, false-positive rates, predictive values and calibration. It should also examine whether information is more often missing or less reliable for particular groups. The choice of measures should reflect the possible harms, rather than selecting only those that make differences look small.

Small groups may have too few outcomes for precise estimates. Wide uncertainty should not be mistaken for evidence that performance is equal. Broad categories can also hide differences within groups, while intersecting characteristics may reveal problems that separate analyses miss. Sensitive information used for evaluation needs appropriate privacy and access safeguards.

#Beyond a fairness score

Fairness has several meanings, and common statistical criteria cannot always be satisfied together, especially when outcome frequencies differ. Matching one error rate does not prove that a system is fair overall. The importance of a difference depends partly on the action it triggers and who bears the consequences.

A broader assessment considers access, workload, patient experience and whether the system improves or worsens existing inequalities. Relevant perspectives include those of affected communities and clinical teams. Changes intended to reduce disparities need evaluation for unintended effects. No single score can replace transparent reasoning about benefits, burdens and acceptable trade-offs.

#Common misunderstandings

A model that performs well overall does not necessarily work equally well for everyone. Strong average results can hide missed diagnoses or unnecessary alerts in smaller groups. However, a difference between groups does not, by itself, explain why that difference exists or how to address it.

Fairness also does not mean making every performance measure identical. Different measures capture different problems, and improving one can sometimes worsen another. The clinical consequences matter: a missed serious condition and an unnecessary follow-up test are not interchangeable harms.

Another misunderstanding is that checking a few demographic categories proves a system is fair. Broad categories can hide important differences within groups, while small samples can make estimates uncertain. People may also belong to several overlapping groups.

Finally, fairness is not a one-time certification. Changes in patients, services or working practices can alter performance. Evaluation needs to consider who receives the technology’s benefits, who faces its burdens and what happens after a result.

#Questions worth asking a clinician

• Was this system tested in patients like me, including my age, sex, ethnicity, health conditions and language?
• Does the system miss diagnoses or raise false alarms more often for some patient groups?
• Could missing or inaccurate information in my medical record make the system’s recommendations less reliable for me?
• Could language barriers, disability, cost or limited internet access prevent me from benefiting from this system?
• How will you check the system’s recommendation and protect me from harm if it performs poorly for people like me?