Confounder Hunters: How LLM Agents are Rewriting the Rules of Causal Inference

TL;DR for operators

Clinical analytics teams already know the unpleasant truth: observational data is cheap, rich, and biased in ways that do not politely announce themselves. The paper behind this article proposes a way to make that bias-hunting process less artisanal. Instead of asking experts to manually inspect every causal-tree rule, the framework lets causal trees segment patients, asks medical LLM agents to suggest plausible confounders using decomposed prompting plus retrieval, sends those suggestions through expert validation, then recursively focuses on samples whose treatment-effect estimates still have wide confidence intervals.¹

The operational lesson is simple, with the usual caveat that “simple” is where expensive mistakes like to hide: the LLM is not the causal estimator. It is a confounder scout. Treatment effects are still handled by causal-tree machinery and evaluated through confidence intervals. That distinction matters because the system’s value is not magical causal reasoning. It is cheaper, faster, more structured triage of where human experts should look.

On Taiwan National Health Insurance Research Database data for Acute Coronary Syndrome, the authors test the framework on two treatment settings, BET and ACE. After iterative refinement, the reported 95% confidence-interval widths fall below those of Causal Forest and Generalized Random Forest baselines. For example, with llama3-Med42, BET width drops from 0.260 at iteration 0 to 0.133 at iteration 3, while ACE drops from 0.246 to 0.141. With Palmyra-Med, BET similarly reaches 0.133, and ACE reaches 0.132 by iteration 2.

That is useful evidence, but not a licence to declare causal victory. Narrower intervals suggest more stable and precise estimates under the framework’s assumptions. They do not prove every confounder has been found, nor that the resulting effect estimates are unbiased in the clinical sense. For business use, the right interpretation is narrower: this is a workflow for prioritising difficult subgroups, discovering plausible clinical confounders, and reducing manual review burden. Boring? Perhaps. Valuable? Usually the same thing, once real money and clinical risk enter the room.

The problem is not estimating effects. It is knowing what you forgot to adjust for

Causal inference in healthcare usually starts with an attractive bargain. Randomised controlled trials are expensive and slow. Observational datasets are already sitting there, full of diagnoses, treatments, demographics, and outcomes. Surely one can extract useful evidence from them.

One can. Carefully.

The danger is confounding. If patients receiving a treatment differ systematically from patients who do not, then the treatment effect may partly reflect those pre-existing differences. In Acute Coronary Syndrome, that is not a footnote. Comorbidities, medication history, disease severity, and clinical pathways can all distort the apparent relationship between treatment and outcome.

Traditional causal machine learning methods try to manage this by modelling heterogeneous treatment effects. Causal trees are especially attractive because they produce interpretable subgroup rules. A leaf in the tree might correspond to a clinically meaningful patient segment, and each segment receives its own estimated treatment effect. This gives analysts something they can inspect.

The weakness is equally obvious. A tree can only split on what it sees. It may produce clean-looking subgroup rules while missing clinically relevant confounders. It is also sensitive to data perturbations: small changes in the training set can alter the tree structure and the resulting estimates. In medical analytics, “interpretable but unstable” is not the upgrade brochure promised.

The paper’s central move is to keep the interpretability of causal trees but add an agentic review loop around them. The LLM does not replace the estimator. It reads the rules, decomposes their clinical meaning, retrieves medical knowledge, proposes confounders, and helps decide where the statistical pipeline should look again.

That is the mechanism worth understanding.

The loop starts with causal trees, not chatbots

The framework begins conventionally. A causal tree partitions observational data into subgroups based on heterogeneous treatment effects. Each leaf corresponds to a rule-defined patient segment and an estimated conditional average treatment effect.

This first tree creates the causal “state space” for the agent. The agent is not handed a vague instruction such as “find bias.” It receives explicit tree rules and subgroup descriptions. Those rules are then converted into reasoning tasks.

The authors use decomposed prompting: a complex causal question is broken into smaller subqueries. For a subgroup rule, the agent may ask how a specific covariate affects the clinical outcome, whether it interacts with the treatment, or whether it might explain subgroup heterogeneity. This matters because single-shot prompting is a fairly heroic way to do clinical reasoning. Heroism is a poor systems design principle.

The agent then retrieves domain knowledge. The paper describes a vector database preloaded with authoritative Acute Coronary Syndrome textbooks. If retrieval does not provide enough useful context, the agent can use external tools such as PubMed. The retrieved material is used to ground the agent’s causal reasoning and reduce unsupported confounder suggestions.

The output is a candidate set of confounders. These are not automatically treated as truth. They go to expert review and then back into the causal estimation pipeline.

The workflow therefore has four control points:

This is the right level of ambition. The system automates hypothesis generation and triage. It does not automate clinical judgement. That is not a limitation to apologise for; it is the reason the workflow is usable.

Confidence intervals become the routing signal

The clever part of the framework is not merely asking an LLM for confounders. Everyone can ask a model for a list of plausible medical variables and receive something that sounds like a ward round performed by autocomplete.

The more interesting move is how the paper decides where to iterate.

The authors use confidence-interval width as a signal of estimation stability. They train multiple causal trees on bootstrap samples and examine the distribution of estimated treatment effects for each sample. Narrow intervals suggest relatively consistent estimates across bootstraps. Wide intervals suggest instability.

The framework then separates samples into two groups. Stable samples, whose confidence-interval widths fall below the threshold, are filtered out. Unstable samples are retained and sent into another round of causal-tree partitioning and agentic confounder discovery. In other words, the model does not keep reworking the whole dataset. It focuses attention on the parts of the population where estimates remain uncertain.

That makes the process look less like a single causal model and more like a Mixture-of-Experts workflow. Each causal tree specialises in a subset of samples. Later iterations handle the cases that earlier trees could not stabilise. The final treatment-effect estimate for a sample is assigned by tracing back through the refinement path to the appropriate subgroup.

For an operator, this is the practical insight: uncertainty is not just an error bar at the end of the report. It becomes a routing mechanism inside the workflow.

That is a useful design pattern beyond this specific clinical case. In insurance risk modelling, policy evaluation, pharma safety surveillance, and hospital operations, there are many settings where the analyst does not need an AI system to “solve causality.” The analyst needs a disciplined way to ask: which records are still suspicious after the first round of adjustment?

The medical agents found plausible confounders, not hidden universal truths

The paper tests several medical LLMs: llama3-Med42-70B, Palmyra-Med-70B-32k, Meditron-70B, and Llama3-OpenBioLLM-70B. The authors note that institutional privacy requirements pushed the experiments onto on-premise infrastructure using four NVIDIA RTX 4500 Ada GPUs.

The agents identify confounders across iterations for two treatment settings. In the BET experiments, model outputs commonly include hypertension, congestive heart failure, atrial fibrillation, coronary artery disease, diabetes mellitus, cerebrovascular disease, and chronic kidney disease. In the ACE experiments, outputs include diabetes, hypertension, chronic obstructive pulmonary disease or COPD-related variables, chronic kidney disease, congestive heart failure, cerebrovascular disease, coronary artery disease, atrial fibrillation, and gout.

The exact model-to-model differences matter less than the pattern. The agent is not discovering a single secret variable that old statistics somehow missed. It is producing clinically plausible confounder candidates that vary by treatment setting, model, and iteration. That variation is both a strength and a warning.

It is a strength because subgroup-specific confounding is exactly the problem. Different patient slices may need different adjustment logic. A rigid one-size covariate set may hide clinically meaningful heterogeneity.

It is a warning because LLM outputs are not stable by default. Two medical models may surface overlapping but not identical confounder sets. This makes expert validation and audit trails non-negotiable. The system should log which rules produced which prompts, which retrieved documents were used, which confounders were suggested, and which ones were accepted or rejected.

Without that trace, the workflow becomes a more elaborate version of “the model said so.” That is not causal inference. That is theatre with a GPU budget.

The evidence: narrower intervals after refinement

The main quantitative evidence is the comparison of average 95% confidence-interval widths on the testing set. The authors compare their iterative framework with Causal Forest and Generalized Random Forest baselines.

Two details deserve attention.

First, the method is not better at iteration 0. For BET, the initial width of 0.260 is worse than both Causal Forest and Generalized Random Forest. The value appears after the loop starts doing its job. The business takeaway is not “use causal trees plus LLMs.” It is “use uncertainty-guided refinement to decide where agentic causal review is worth spending effort.”

Second, confidence-interval width is a precision and stability measure. It tells us that estimates became less uncertain under the procedure. It does not independently prove the true treatment effect was recovered. A model can be confidently wrong, especially if the missing variable is absent from the data and absent from the retrieved knowledge. The paper’s language sometimes gestures toward “unbiased” estimation, but an operator should translate that carefully: the workflow provides evidence of reduced estimation uncertainty after iterative confounder adjustment, not a courtroom certificate that all bias has been eliminated.

Figure 2 is a triage story, not just a convergence story

The paper also tracks unstable samples across iterations. The number of unstable samples falls over time, while stable samples accumulate. At the end, 169 BET samples and 446 ACE samples still exceed the confidence-interval thresholds across all iterations.

This is operationally important. A failed-to-stabilise sample is not merely a modelling nuisance. It is a queue item for expert review.

In a clinical analytics department, the worst process is often the flat review process: every rule, every subgroup, every suspicious pattern gets dumped on the same limited experts. The paper’s workflow suggests a different operating model. Let the statistical system and the agentic confounder scout reduce the search space. Then send the persistent failures to humans.

That does not remove expert judgement. It gives expert judgement better targets.

The remaining unstable cases may reflect unobserved confounders, incomplete covariates, insufficient sample size, noisy treatment assignment, or clinical complexity that the current representation cannot capture. The paper interprets them as possible signals of unobserved confounding. That is plausible, but should be treated as a diagnostic hypothesis rather than a final diagnosis.

The distinction matters. In production, one would not say, “These 446 ACE samples contain unobserved confounders.” One would say, “These 446 samples remain unstable after the current adjustment loop and should be reviewed for missing clinical structure, coding artefacts, subgroup sparsity, or confounding.”

Less dramatic. More useful.

What this changes for healthcare analytics teams

The paper’s contribution is best understood as workflow architecture.

Most organisations do not suffer from a shortage of causal models. They suffer from a shortage of trusted, reviewable, domain-aware causal workflows. The bottleneck is not just computation. It is the translation layer between model output and expert judgement.

This framework suggests a practical division of labour:

For hospitals, this could support comparative effectiveness studies where clinical teams need interpretable subgroup-level estimates. For insurers, it could help identify patient segments where treatment pathways appear beneficial or risky but confounding remains unresolved. For pharma safety and real-world evidence teams, it could make observational studies more reviewable by forcing the model to expose the confounder hypotheses behind subgroup adjustments.

The commercial value is not that a vendor can say “LLM-powered causal inference” in a pitch deck, although one assumes someone is already warming up the slide. The value is process compression: fewer expert-hours spent scanning every subgroup rule, more attention directed to unstable or clinically suspicious segments, and clearer documentation of why particular confounders were considered.

That is meaningful ROI if the organisation already has the data infrastructure, clinical review capacity, and governance discipline. Without those, the agent just produces more confident paperwork.

Where the paper is strong, and where operators should stay sceptical

The paper is strongest in three places.

First, it treats LLMs as agents embedded in a statistical pipeline rather than as standalone answer machines. That is the right pattern for high-stakes analytics. The model’s job is bounded: interpret rules, retrieve evidence, suggest confounders, and assist prioritisation.

Second, it connects interpretability and uncertainty. The causal-tree structure gives humans rules they can inspect. Confidence-interval width gives the system a measurable reason to continue refinement. This pairing is more practical than simply presenting a black-box heterogeneous treatment-effect model and hoping clinicians enjoy the mystery.

Third, it reports a real clinical-data application rather than a purely synthetic benchmark. Taiwan’s NHIRD offers longitudinal medical claims data, and the Acute Coronary Syndrome setting is clinically plausible for treatment-effect heterogeneity and confounding.

The boundaries are equally important.

The study focuses on one clinical application. The reported improvements may not transfer to oncology, mental health, hospital operations, health economics, or non-medical policy domains without adaptation.

The evaluation emphasises confidence-interval width. That is useful, but it is not the same as external validation against a known causal truth. In observational healthcare data, the truth is precisely the slippery object.

The method depends on the quality of candidate covariates, retrieval sources, prompts, model behaviour, and expert validation. If the relevant confounder is not recorded, not retrievable, or not recognised by experts, the loop cannot conjure it into existence. Causal inference remains stubbornly unimpressed by enthusiasm.

The paper also leaves practical governance questions open. How consistent are confounder suggestions across random seeds or prompt variants? How should rejected confounders be documented? What threshold should trigger escalation to human review? How expensive is local deployment at scale? How does the workflow perform when clinical notes, rather than structured claims variables, become central? These are not objections to the idea. They are the questions a deployment team should ask before moving from prototype to decision support.

The executive interpretation: automate the search, not the judgement

The easiest way to misread this paper is to say: “LLM agents can do causal inference.”

They cannot, at least not in the way that sentence implies.

A better reading is this: LLM agents can help causal inference workflows become more semantically aware. They can read subgroup rules, connect them to medical knowledge, suggest plausible confounders, and help route unstable samples through additional review. The causal estimates still need statistical machinery. The confounder decisions still need domain validation. The results still need humility.

For business leaders, that is actually good news. Fully automated causal inference would be difficult to trust and harder to govern. A bounded assistant that reduces expert review burden, documents candidate confounders, and highlights unstable subgroups is more deployable.

The paper points toward a useful middle ground between two bad extremes. On one side is manual causal review, slow and expensive. On the other is black-box automation, fast and occasionally reckless. The proposed framework sits between them: interpretable trees, agentic confounder discovery, retrieval-grounded reasoning, confidence-interval routing, and human validation.

That is not a revolution with fireworks. It is a better assembly line for a difficult analytical job.

In healthcare, that may be exactly the kind of progress that survives contact with reality.

Cognaptus: Automate the Present, Incubate the Future.