When Predictions Persuade: The Hidden Causal Risks of AI Decision Support

A prediction looks harmless when it is presented as “just information.”

A loan officer sees a default-risk score. A doctor sees a survival estimate. A welfare caseworker sees a predicted probability of program success. The model does not press the button. The human still decides. Everyone in the room can therefore relax, at least until the audit committee arrives with coffee and regrettable questions.

The paper “2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support” by Otto Nyberg, Fausto Carcassi, and Giovanni Cinà makes that comforting story harder to maintain.¹ Its central point is not that prediction models are biased, inaccurate, or badly optimized, although those problems certainly exist elsewhere. The sharper point is that even a good prediction model can change a user’s beliefs in ways that lead to worse downstream decisions.

That is the unpleasant part. The model may be “decision support,” but the support is not passive. A prediction is evidence. Evidence updates beliefs. Beliefs change causal judgments. Causal judgments determine action. Action determines outcome.

The paper calls this the 2-Step Agent framework. The name is modest. The implication is not.

The real object of study is not the prediction; it is the user after seeing the prediction

Most AI decision-support evaluation begins with the model. How accurate is the prediction? Does it generalize? Is it calibrated? Does it perform equally across groups? These questions are necessary, but this paper asks a different one:

What happens inside a rational decision maker after they observe a model prediction?

That shift matters because a prediction used for decision support is not merely a number attached to a case. It is also a clue about the world that produced the model. A user may reason, explicitly or implicitly, about the training population, the historical treatment policy, the outcome process, and the model’s relationship to all of them.

The paper’s running example is medical. An oncologist receives a model prediction that a cancer patient has only two months to live. That prediction alone does not determine the right action. If the model was trained mostly on patients who received minimal treatment, the poor prognosis may suggest aggressive treatment could help. If the model was trained mostly on patients who already received heavy treatment, the same poor prognosis may suggest that more treatment is futile and palliative care is preferable.

Same prediction. Opposite action.

The difference is not model accuracy. It is the decision maker’s belief about what the prediction means.

That is the first mechanism the paper formalizes. A user does not simply read a predicted outcome. The user also interprets the prediction as evidence about the population and historical data behind the model.

The two steps: belief update first, causal decision second

The framework decomposes AI-assisted decision making into two linked steps.

First, the agent observes a new case and, when decision support is available, the model’s prediction for that case. The agent then performs a Bayesian update over an internal model of the world. This internal model includes beliefs about the historical data-generating process, the predictive model, and the training data from which the model was learned.

Second, the agent uses the updated beliefs to estimate the causal effect of possible actions for the new case. The action is chosen when the estimated conditional average treatment effect, or CATE, crosses a decision threshold.

A simplified version of the mechanism looks like this:

This is the paper’s core contribution. It makes the decision maker part of the computational model. The user is not a vague “human in the loop,” that magical compliance fairy often invoked when automation becomes legally or ethically inconvenient. The user is a reasoning system with priors, uncertainty, and a decision rule.

The important distinction is this: the prediction model estimates something observational, while the decision maker needs something interventional.

A treatment-naive prediction model estimates something like:

$E(Y \mid X)$

But the decision maker needs to compare outcomes under possible actions:

$E(Y \mid X, do(A = a_1)) - E(Y \mid X, do(A = a_0))$

The paper’s agent does not blindly confuse these two quantities. Instead, the agent uses the prediction as evidence and then estimates the causal effect through its own internal causal model. That makes the result stronger, not weaker. The paper is not saying, “humans are irrational, so prediction tools are risky.” It is saying that even a sophisticated Bayesian reasoner can be pulled into bad decisions if the prior assumptions used to interpret the prediction are misaligned.

A worse result obtained by an irrational user is not news. A worse result obtained by a rational user is more interesting. And less convenient.

Why a prediction can correct one belief but corrupt another

The paper’s simulations use a simplified medical-style setting: one continuous covariate, interpreted as body weight; one continuous treatment, interpreted as dosage; and one continuous outcome, interpreted as months of survival. The true data-generating process makes treatment beneficial on average. The model used for decision support is a slope-only linear regression that predicts outcome from the covariate, without directly modeling treatment.

The agent chooses between two dosages, 10 and 20. The higher dosage is selected only if the estimated CATE exceeds a threshold of 5. In plain English: the agent needs to believe the higher dose improves expected outcome by at least five months before choosing it.

The experiments then vary the agent’s prior beliefs one at a time. This is not a field trial. It is a mechanism laboratory. The purpose is to see how a single wrong belief can change the effect of decision support.

The paper examines several kinds of prior belief:

The result is asymmetric. If the agent is wrong only about treatment effect while other beliefs are aligned, observing the model prediction can help. The decision-support signal pushes the agent’s CATE estimate closer to the true value, and downstream outcomes improve.

That is the happy case, and the paper does not hide it. AI decision support can teach the agent something useful. When the surrounding assumptions are right, the model prediction can discipline a mistaken belief about treatment effectiveness.

But the fragile part comes next.

When the agent is wrong about the historical treatment policy, the same prediction can lead to worse action. The agent tries to explain the gap between the model’s prediction and the outcome they expected. If their belief about past treatment is wrong, the explanation may move the causal estimate in the wrong direction. The agent can end up choosing the wrong dosage and producing worse outcomes than without decision support.

A similar pattern appears when the wrong belief concerns the training population’s covariate distribution or the outcome distribution. The model’s prediction introduces new dynamics into the CATE estimate. In some regions, the estimated treatment effect can change substantially in magnitude or even sign. Once the sign flips, the business problem stops being “imperfect support” and becomes “confident misdirection.”

The paper’s Figure 3 is the main evidence for this mechanism. The top row tracks how the agent’s CATE estimate changes with and without ML decision support. The bottom row tracks the downstream outcome after the agent acts. The useful reading is not “orange line good” or “blue line bad.” The useful reading is conditional:

This table is important because it prevents a common misreading. The paper is not presenting one universal “AI helps” or “AI harms” result. It is showing a mechanism that can move in either direction depending on the agent’s priors.

A predictive model can be beneficial when it helps the decision maker correct the right belief. It can be harmful when it pushes the decision maker to revise the wrong belief.

That is the small hinge on which the whole door turns.

The hidden business variable is user belief alignment

For enterprise AI, the usual deployment checklist asks whether the model is accurate, documented, secure, compliant, and integrated into workflow. This paper adds another variable:

Are the user’s beliefs about the model’s training world aligned with the world the model actually came from?

That question is more operational than it sounds.

A credit analyst may need to know whether historical approvals were conservative or aggressive. A hospital specialist may need to know whether historical outcomes reflect standard care, under-treatment, or over-treatment. A claims manager may need to know whether past interventions were assigned by strict protocol, staff discretion, budget pressure, or regional habit. A sales team using churn predictions may need to know whether the training data includes prior retention campaigns, discounts, or account-manager interventions.

The model prediction is not self-explanatory. It is a compressed trace of the training environment. Users will unpack that trace using whatever story they carry into the workflow.

Usually, that story is not written down.

This is where the paper becomes practical. The authors emphasize documentation and training, but the deeper implication is that documentation should not only describe the model. It should describe the historical decision process behind the model.

A useful documentation package for decision support should answer questions like:

This is not decorative governance. It directly affects the mechanism in the paper. Better documentation changes the priors with which users interpret the prediction. Training is not merely “how to click the dashboard.” Training is prior engineering.

Yes, that phrase sounds like it escaped from a Bayesian startup pitch deck. Still, it is the right idea.

“Human in the loop” is not a safety guarantee

The paper’s misconception target is clear: keeping a human in the loop does not automatically make predictive decision support safer.

The reason is that the human is not just approving or rejecting a machine suggestion. The human is learning from the machine. If the user treats the prediction as evidence about the historical population, the model can reshape the user’s causal beliefs.

That matters especially when the prediction model is treatment-naive. A treatment-naive model can be useful for forecasting outcomes, but it does not by itself answer “what should we do?” It may estimate how patients historically fared, how applicants historically defaulted, or how customers historically churned. But a manager needs to know how outcomes change under alternative interventions.

That gap is where belief does the work.

Without decision support, the agent uses prior beliefs to estimate the treatment effect. With decision support, the prediction changes those beliefs before the action is selected. If the belief update is well-directed, the decision improves. If it is misdirected, the decision worsens.

For business leaders, this creates a more precise way to evaluate deployment risk:

This is also why “just show the score and let professionals decide” is not a serious risk-control strategy. Professionals will decide, but they will decide using beliefs. If deployment changes those beliefs in an untested way, oversight becomes another channel of model influence.

The technical appendix is not decoration; it explains why this framework is hard to compute

The paper’s third contribution is more technical: it derives a sufficient-statistics reduction for Bayesian inference through latent training-data plates in the linear-model case.

That sounds like a paragraph written to make executives close the browser. It is worth translating.

The agent observes a prediction from a model trained on historical data. But the agent does not observe the full training dataset as a concrete object in the inference problem. The training data become latent variables: many interchangeable past cases that could have produced the model parameter. To update beliefs properly, the agent must reason through this hidden training process.

Naively, this is computationally expensive and numerically unstable. The appendix shows that, for the paper’s simple linear-regression setting, the full plate of repeated training observations can be collapsed into a smaller set of sufficient statistics. In practical terms, the authors reduce the inference burden by replacing many latent historical observations with lower-dimensional summary variables that preserve the needed information for the model parameter.

This is not the headline business insight, but it matters for the research program. If we want to simulate how users learn from model predictions, the inference problem has to be computationally tractable. Otherwise the framework remains elegant and unusable, the academic equivalent of a glass staircase.

The appendix also includes sanity checks: comparing explicit-plate and sufficient-statistics formulations, verifying posterior movement under controlled observations, checking convergence toward real generating parameters under many observations, and examining induced correlations among parameters. These are not additional business claims. They are implementation checks supporting the machinery used in the main simulations.

That distinction matters. The appendix strengthens confidence that the simulated agent is doing what the framework says it is doing. It does not make the simplified medical setting magically representative of hospitals, banks, courts, or welfare agencies.

What Cognaptus infers for enterprise AI deployment

The paper directly shows a formal mechanism and simulation results: AI decision support can improve or worsen downstream outcomes depending on how a Bayesian agent updates beliefs after observing a prediction. It also shows that a single misaligned prior can be enough to produce harm in the simulated setting.

For business use, Cognaptus would translate that into four deployment practices.

First, evaluate decision support at the action level, not only the prediction level. A model that predicts well may still induce worse choices if users interpret the prediction causally in the wrong way. Pre-deployment testing should measure how recommendations or risk scores change decisions, not merely whether users find the interface helpful.

Second, document historical policy. Users need to know what past actions generated the training data. This is particularly important when the model predicts outcomes under historical behavior but is used to guide future behavior.

Third, train users on what the prediction is and is not. A risk score is not automatically a treatment-effect estimate. A predicted outcome is not automatically a recommendation. If the model is treatment-naive, the user must understand that limitation in operational terms, not as a footnote buried beneath compliance language.

Fourth, simulate user belief scenarios before deployment. The paper offers a formal version of this idea: vary user priors and observe how AI decision support changes action and outcome. Real businesses do not need to copy the exact Bayesian machinery for every deployment. But they should stress-test plausible user interpretations. For example: what happens if users believe the training population was heavily treated? What happens if they assume the historical policy was conservative? What happens if they think the score is causal when it is merely predictive?

A practical deployment review could use a table like this:

The ROI relevance is not “better AI ethics,” although that is nice for annual reports. The ROI relevance is avoiding expensive decision-support systems that look accurate in validation and then quietly degrade operational outcomes because users learn the wrong lesson from the score.

Boundaries: what this paper does not prove

The paper’s limitations are important, and they are refreshingly specific.

The experiments use a simple data-generating process: normal distributions, one confounder, a simplified treatment-outcome structure, and a linear treatment-naive prediction model. The agent chooses between two dosage levels. The agent is Bayesian and, in that sense, idealized. Real users are messier. They rely on heuristics, institutional routines, incentives, liability concerns, and occasionally vibes wearing a badge that says “experience.”

The simulation also does not establish field effect sizes. It shows a mechanism under controlled assumptions. It does not tell a hospital or lender exactly how much outcome degradation to expect from a specific deployed model.

Another boundary is that the model used for decision support is not causal. The authors explicitly leave causal decision-support models for future work. That matters because a system designed to estimate treatment effects may change the interaction pattern. It would not eliminate user-belief issues, but it would alter what the user is being asked to infer.

Finally, the framework currently focuses on a single interaction with a new case. Future work could examine repeated learning, heterogeneous treatment effects, more complex model classes, and agents with different internal models. These extensions are not decorative. In real enterprise settings, users repeatedly interact with decision-support tools and may adapt over time. The first week of deployment and the sixth month of deployment are rarely the same animal.

The useful warning is not “AI is dangerous.” It is narrower and better.

The strongest reading of the paper is not that decision support should be avoided. The paper actually shows settings where ML-DS improves decisions by correcting a mistaken treatment-effect belief. The better warning is more precise:

Prediction tools change decisions through belief updates, and those updates depend on what users believe about the training world.

That gives managers a sharper deployment question. Not “Is the model accurate?” Not “Is there a human in the loop?” Not even “Did we explain the dashboard?”

The question is:

After seeing the prediction, what will the user believe has been learned about the causal effect of action?

If the answer is unclear, the system is not fully evaluated.

AI decision support is often sold as a way to combine machine intelligence with human judgment. This paper reminds us that combination is not automatically complementarity. Sometimes the prediction improves judgment. Sometimes it persuades judgment into the wrong causal story.

The machine may only output a number. The human supplies the interpretation. The outcome comes from the pair.

That is where the risk hides.

Cognaptus: Automate the Present, Incubate the Future.