Opening — Why this matters now
Large language models are quietly moving from clerical assistance to clinical suggestion. In emergency departments (EDs), where seconds matter and triage decisions shape outcomes, LLM-based decision support tools are increasingly tempting: fast, consistent, and seemingly neutral. Yet neutrality in language does not guarantee neutrality in judgment.
This paper interrogates a subtle but consequential failure mode: latent bias introduced through proxy variables. Not overt racism. Not explicit socioeconomic labeling. Instead, ordinary contextual cues—how a patient arrives, where they live, how often they visit the ED—nudging model outputs in clinically unjustified ways.
Background — Context and prior art
Emergency Severity Index (ESI) triage already struggles with accuracy. Large-scale studies show that roughly one-third of high-acuity cases are mis-triaged, with well-documented racial disparities. AI systems are often proposed as a corrective force. However, most bias audits in healthcare AI focus on direct attributes (race, gender, age) or aggregate subgroup performance.
What is less explored is how indirect signals act as stand-ins—proxies that correlate with social or economic disadvantage and quietly shape predictions. Prior literature has acknowledged proxy bias in principle, but rarely tested it in a controlled, patient-level, language-driven setting.
Analysis — What the paper does
The study introduces a clean and surprisingly powerful evaluation framework.
Core idea: If an LLM truly understands clinical triage, then adding non-clinical context should not systematically change ESI predictions.
To test this, the author:
-
Identified 32 patient-level proxy variables (e.g., insurance status, arrival mode, neighborhood income, social isolation).
-
For each proxy, created paired qualifiers:
- Positive (socially advantaged framing)
- Negative (socially disadvantaged framing)
-
Injected these qualifiers into otherwise identical ED visit scenarios drawn from MIMIC-IV(-ED) data.
-
Asked an LLM (gpt-4o-mini) to assign ESI scores under three conditions:
- Baseline (no qualifier)
- Positive qualifier
- Negative qualifier
Critically, all qualifiers were manually scrubbed to avoid introducing legitimate ESI-relevant information. Any observed shift, therefore, reflects model sensitivity to context—not clinical reasoning.
Findings — What actually changed
The results are uncomfortable.
1. Polarity-dependent bias
For many proxies, direction mattered. Negative qualifiers (e.g., uninsured, recent incarceration, high-crime area) pushed the model toward higher perceived acuity, while positive framings pulled acuity down. This mirrors classic social bias pathways—only now encoded in token probabilities rather than human intuition.
2. Polarity-independent bias (the stranger one)
More troubling: some proxies shifted ESI in the same direction regardless of framing. Whether a qualifier was positive or negative, the mere presence of certain tokens altered severity estimates.
This suggests the model is not reasoning over meaning at all—it is reacting to statistical residue. Tokens become triggers, not concepts.
3. Negligible effects
A minority of proxies showed little influence, likely reflecting either balanced training data or weak token associations. These are the exceptions, not the rule.
Summary table
| Bias Type | Mechanism | Risk |
|---|---|---|
| Polarity-dependent | Social framing alters acuity | Discrimination across cohorts |
| Polarity-independent | Token presence overrides semantics | Systematic mis-triage |
| Negligible | No consistent shift | Low immediate risk |
A concrete example — Ambulances and race
The paper closes the loop with population data.
White patients, holding acuity constant, are more likely to arrive by ambulance than Black patients. The LLM interprets ambulance arrival as higher severity. Result: identical clinical presentations, different perceived urgency.
No explicit race token is required. The bias propagates through infrastructure, access, and habit—then emerges at inference time as a “reasonable” adjustment.
This is not malicious AI. It is statistically obedient AI.
Implications — What this means for deployment
Three implications stand out:
- Fairness audits must move beyond labels. Removing race from inputs does not remove race-linked behavior.
- Semantic alignment is not guaranteed. LLMs can satisfy instruction-following while still misusing context.
- Clinical AI needs proxy-aware guardrails. Especially in high-stakes, resource-constrained settings.
In practice, this argues for:
- Proxy-variable stress testing
- Contextual ablation during validation
- Explicit separation between clinical signals and social descriptors
Conclusion
This paper does not argue against LLMs in healthcare. It argues against naïve trust.
Bias is no longer just about what models are told. It is about what they notice, what they overweight, and what they quietly carry forward from the world that trained them.
In emergency medicine, where triage is destiny, hidden signals are not a technical detail. They are a governance problem.
Cognaptus: Automate the Present, Incubate the Future.