A hospital uses an AI system to predict which patients will be readmitted within the next 30 days.

The model achieves an accuracy of 92 % in tests and was trained on historical data.

• Patients from wealthy areas are classified as “high risk” less often.
• Patients from poorer areas significantly more often.

An analysis shows: The model heavily relies on the number of previous doctor visits.

People from wealthier areas visit doctors more frequently for preventive care. People from poorer areas often go only when problems become serious.

It is statistically plausible, but structurally unfair.

The AI uses “doctor visits” as a proxy for health risk. However, this proxy strongly depends on access to healthcare.

A system that “learns from data” can therefore reinforce existing inequalities — even without intention.

Key point:
The discrimination does not originate from the machine itself, but from historical reality, proxy variables, and goal definitions.

Who is responsible?
The organization operating the system must define fairness criteria and implement monitoring and safeguards.