Mind Games: How LLMs Subtly Rewire Human Judgment

TL;DR for operators

When an LLM summarises a review, policy memo, support ticket, medical note, or news item, the operational question is not only “Did it get the facts right?” The sharper question is: did it change what the user is likely to believe, prioritise, or buy?

The paper behind this article studies exactly that problem. It treats LLM-generated content as a decision interface and measures three ways the interface can quietly bend human judgment: changing the sentiment frame of the source, overemphasising the beginning of the source, and fabricating confident answers for events beyond the model’s knowledge cutoff.¹

The headline numbers are not cosmetic. Averaged across the tested models, the authors report framing changes in 26.42% of summaries, primacy bias in 10.12% of summaries, and user exposure to hallucinated post-cutoff content in 60.33% of instances. In a human-subject product-choice study, participants selected the manufacturer associated with positively reframed LLM summaries 83.7% of the time, compared with 52.3% when shown the original neutral or negative reviews. Willingness to pay rose by 4.5%.

For businesses, the lesson is uncomfortable but useful. Summary quality assurance cannot stop at factuality checks. It needs sentiment preservation tests, coverage balance tests, temporal boundary tests, and behavioural-risk review for contexts where generated text feeds decisions. A summary is not a smaller document. It is an edited experience. The editor is just pretending to be a compression algorithm.

The ordinary product-summary screen is already a decision interface

Imagine a shopper comparing two products. They do not read every review. Of course they do not. That would be spiritually admirable and commercially absurd. They skim the generated summary, glance at the rating, and move on.

Now suppose the original review says: “The device is useful, but the battery is poor, the setup is clumsy, and I am returning it.” The LLM summary says: “The device is useful and offers good customisation, though storage and battery life could be better.” No explicit lie. No dramatic hallucination. Just a slightly kinder lens.

That is the problem this paper makes measurable. The model does not need to fabricate an entire product feature to change a decision. It can alter the frame, omit the middle, overemphasise the opening, or answer with inappropriate confidence when it should admit temporal ignorance. The user receives processed evidence, then behaves as if they saw the evidence itself.

This is why the paper is more interesting than another “LLMs have bias” entry in the already crowded museum of obvious warnings. Its contribution is not that models are imperfect. We knew. The contribution is that the authors operationalise a more specific failure: LLM outputs can induce human cognitive bias by changing the informational conditions under which people decide.

That distinction matters. A biased model output is an engineering defect. A biased model output that changes user behaviour is a product-risk mechanism.

The paper studies bias as content alteration, not model personality

The authors focus on three cognitive-bias pathways. Each begins with a text transformation and ends with a plausible human decision effect.

This framing avoids a common but lazy interpretation: “summarisation is just compression.” It is not. Summarisation selects, weights, paraphrases, and orders evidence. Those are editorial acts. The only reason they feel less editorial inside an LLM interface is that the interface has excellent manners.

The paper uses six models across five families: GPT-3.5-turbo, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct, Phi-3-mini-4k-Instruct, Qwen3-4B-Instruct, and Gemma-3-27B-IT. For summarisation, the authors test Amazon Reviews and MediaSum news interviews. For temporal hallucination, they construct NewsLensSync, a self-updating dataset of real news items and falsified counterparts designed to sit beyond the models’ knowledge cutoffs.

The design is useful because it separates three tasks that are often muddled together:

The important move is that the authors do not treat bias as an abstract moral stain on the model. They treat it as an observable change between input and output.

That is the correct unit of analysis for operators. You cannot govern “bias” in the abstract. You can test whether a generated summary preserved the source sentiment, covered the full context, and avoided making confident claims outside its knowledge boundary.

Mechanism 1: The model changes the frame before the user sees the evidence

The framing test is straightforward. Classify the source text as positive, negative, or neutral. Generate a model summary. Classify the summary. Count cases where the label changes.

Across the tested models and datasets, the paper reports an average framing-change rate of 26.42%. The spread is important. GPT-3.5-turbo and the Llama models sit around 14.5% to 16% in the reported aggregate framing-change fraction. Phi-3-mini, Qwen3-4B, and Gemma-3-27B show higher rates, reaching between 26.5% and 45.9% depending on model and dataset.

The pattern is not uniform across domains. In MediaSum news interviews, many shifts occur between neutral and negative. In Amazon Reviews, one common shift is from positive to neutral for several models, while Qwen3 and Gemma show more neutral-to-positive movement than the others. That matters because the operational concern is not merely “the model is too positive” or “the model is too negative.” The deeper concern is frame instability.

A frame-stable summariser should preserve the decision-relevant tone of the source. If the review is mixed but ultimately rejecting the product, the summary should not polish it into mild satisfaction. If the interview is neutral but contains criticism, the summary should not inflate the criticism into the main emotional signal. In commercial interfaces, that difference can become conversion. In compliance contexts, it can become risk under-reporting. In internal decision memos, it can become a false sense of consensus. Very efficient. Also not ideal.

The paper’s example of a Samsung tablet review makes this tangible. The original text contains praise, frustration, practical complaints, and the final decision to return the product. The generated summary preserves several complaints but presents them in a cleaner, more compressed negative frame. In other cases, the direction can flip toward positivity. Either way, the issue is not whether the model copied every adjective. The issue is whether the user receives the same evaluative posture as the source.

For business deployment, this suggests a simple but underused evaluation layer: compare source sentiment distribution with output sentiment distribution. Not only at the document level, either. Segment-level sentiment preservation is often more useful, because the model may preserve the average while burying the late-stage reversal where the actual decision signal lives.

Mechanism 2: The beginning of the source punches above its weight

The second mechanism is positional. The authors divide each source into three equal segments: beginning, middle, and end. They then compare embedding similarity between the generated summary and each segment. A summary is marked as exhibiting primacy bias when its similarity to the beginning exceeds its similarity to the middle by a defined threshold.

The aggregate primacy-bias score is 10.12% across tested models. On Amazon Reviews, individual model scores range from 5.4% to 23.7%. On MediaSum, they range from 4.0% to 17.6%. The appendix reports paired t-tests showing that beginning-summary similarity is significantly higher than middle-summary similarity across all tested models and datasets.

This is the part operators should read slowly. A 10.12% primacy-bias score does not mean “only one in ten summaries care about the beginning.” It means one in ten crosses the authors’ threshold for a particular operational definition of disproportionate beginning-over-middle similarity. The broader table still shows a consistent pattern: summaries tend to align more with the beginning than with the middle or end.

That is exactly where real documents become dangerous. Long reviews, interviews, incident reports, and meeting notes often contain reversals. A customer starts polite, then describes the defect. A manager begins with context, then explains the exception. A physician note opens with history, then documents the acute concern. A policy document starts with aims, then lists constraints. The middle is where inconvenient reality frequently lives.

The model’s tendency to favour early context is therefore not just an academic positional quirk. It can turn into selective exposure. Users see a summary that feels complete because it is fluent and coherent. Meanwhile, the source has been editorially thinned.

The paper’s mitigation results make the puzzle sharper. Weighted Summaries, which splits the source into beginning, middle, and end and assigns token budgets to each, increases overall coverage similarity. For Llama-3-8B on Amazon Reviews, the reported average coverage similarity rises from 0.843 to 0.910. That sounds like a fix until the next metric walks in wearing muddy shoes: the primacy-bias score also rises from 7.0% to 15.1%.

This is not a contradiction. It is a warning. Better coverage on average does not automatically mean better balance under every bias metric. A method can pull in more content from all sections while still producing outputs whose beginning-over-middle imbalance crosses the threshold more often. Evaluation needs multiple dials, not one large green “summary good” button. We regret to inform procurement.

Mechanism 3: The model invents authority when the event is outside its memory

The hallucination test is built around time. The authors evaluate models on real news items and falsified counterparts, before and after the model’s knowledge cutoff. The strict accuracy metric counts a paired item as correct only if the model identifies both the true item and its falsified version correctly.

That paired design is the valuable part. A model that says “true” too often may look good on real news and terrible on falsified news. A model that says “false” too often may look cautious while failing to recognise real events. Strict paired accuracy exposes whether the system can discriminate, not merely whether it has a convenient default answer.

The results are not flattering. Llama-3-8B-Instruct drops from 26% strict accuracy on pre-cutoff data to 21% post-cutoff. Llama-3.2-3B-Instruct drops from 19% to 15%. Phi-3-mini-4k-Instruct drops from 12% to 8%. The paper’s broader aggregate reports user exposure to hallucinated post-cutoff content in 60.33% of instances.

The practical interpretation is not “never ask an LLM about news.” The practical interpretation is more precise: when the model is asked to fact-check claims outside its training freshness, fluent binary answers are a liability unless the system has retrieval, temporal awareness, or explicit uncertainty handling.

This matters for customer support, investor monitoring, compliance screening, policy analysis, and any internal system where users ask, “Has X happened?” A model that cannot know may still answer. That is the ancient software tradition of returning something because null feels rude.

The authors test several hallucination mitigations. Prompt calibration with chain-of-thought improves falsified-news detection for Llama-3-8B but has limited utility for other models. Knowledge Boundary Awareness, which tells the model its cutoff date, improves some falsified-news detection but can reduce other accuracy measures. Epistemic Tagging, where the model must attach high or low confidence to its true/false answer, performs best across the tested models in the authors’ discussion, especially improving strict accuracy for smaller models such as Phi-3-mini.

The operator takeaway is not that confidence tags magically create knowledge. They do not. They create a surface where uncertainty can be routed. A tagged low-confidence answer can trigger retrieval, escalation, source display, or refusal. Without that workflow, confidence labels risk becoming decorative metadata, which is governance theatre with better typography.

The user study turns a text defect into a behavioural result

Many LLM evaluation papers stop at output metrics. This paper goes further by asking whether altered outputs change human decisions.

The human study uses product reviews. The authors first probe Amazon reviews and generate summaries with GPT-3.5. To examine a worst-case scenario, they use GPT-4 to identify product types with the most extreme framing changes, then manually select ten product pairs. In each pair, one product review shifts from negative or neutral to positive after summarisation, while the other maintains neutral framing.

Participants choose between two manufacturers based on either original reviews or summaries. They also report willingness to pay for the selected product. The study recruits 72 Prolific participants, accepts 70 submissions after a comprehension quiz, and uses alternating conditions so participants see both original reviews and summaries across product pairs.

The result is the paper’s most business-readable finding: participants selected manufacturers associated with positively framed summaries 83.7% of the time, compared with 52.3% when those same products were represented by the original neutral or negative reviews. Willingness to pay increased by 4.5% relative to the original product price.

That is not a generic conversion-lift benchmark. Do not put “LLM summaries raise WTP 4.5%” into a sales deck unless you enjoy misleading people efficiently. The products were selected to represent strong framing-change cases. The study is best read as evidence of mechanism plausibility: when summarisation substantially reframes product evidence, user choice and valuation can move with it.

For operators, that is still enough. If generated summaries sit upstream of purchases, applications, risk approvals, treatment options, hiring screens, or escalation decisions, then summary drift is not a mere text-quality issue. It is a behavioural intervention.

The distinction is simple:

This is the article’s central business point: the risk is not that the model has opinions. The risk is that the model edits the evidence before the user forms theirs.

The mitigation results say “pick your failure mode,” not “install prompt and relax”

The authors test eighteen mitigation methods. Some are prompt-level interventions, such as self-awareness prompts and chain-of-thought. Some restructure the input or output process, such as chunked summaries, weighted summaries, attention sorting, and position-invariant shuffling. Some intervene at decoding time, such as weighted token decoding, Mirostat, rejection sampling, self-debias decoding, and local-explanation guards. For hallucination, they test prompt calibration, knowledge boundary awareness, and epistemic tagging.

This section is easy to misread. A casual reader might scan for “best mitigation” and leave with a shopping list. The more useful reading is that mitigation is non-modular. A fix aimed at one bias can worsen another.

For Llama-3-8B on Amazon Reviews, Weighted Token Decoding reduces the framing-change rate from 14.5% to 13.6%, the best framing result in the highlighted table. But it worsens the primacy-bias score from 7.0% to 18.4%. Weighted Summaries improves coverage similarity, but increases the primacy-bias score in the same setting. Position-Invariant Shuffle is diagnostically useful because it tests sensitivity to order, but it can disrupt temporal or narrative structure and degrade other metrics.

This is exactly what should happen when multiple desiderata compete. A summary can be faithful to sentiment, broad in coverage, concise, temporally coherent, and readable. But pushing one property often pulls another. The system designer’s job is not to chant “debiasing” at the model. It is to specify which failure is costliest in the deployment context.

A product-summary system may prioritise sentiment preservation and defect coverage. A legal-summary system may prioritise source-grounded completeness and citation traceability. A clinical note summariser may prioritise not omitting late-emerging adverse details. A news fact-checker must prioritise temporal retrieval and uncertainty escalation. Same model family, different loss function in the real world.

The mitigation table is therefore less a recipe book than a diagnostic map. It says: measure the failure mode you care about, test interventions against multiple metrics, and expect trade-offs. Annoying, yes. Also how engineering works when slogans leave the room.

What operators should test before shipping generated summaries

The paper suggests a practical evaluation suite for any organisation using LLMs to transform source material into decision-facing text.

First, test frame preservation. Compare the sentiment or evaluative stance of source and output. For high-stakes or high-volume systems, do this at both document and segment level. A model that preserves the overall tone while dropping the one paragraph that contains the actual defect is still failing.

Second, test coverage balance. Split the source into meaningful sections, not always equal thirds. For reviews, beginning/middle/end may work. For contracts, use clauses. For medical notes, use clinical sections. For customer support, separate complaint, troubleshooting, response, and unresolved issue. Then measure whether the summary represents each decision-relevant section.

Third, test temporal boundaries. If users ask about current facts, build retrieval into the system or force abstention when no source is available. A static model should not be asked to impersonate a live database. That is not intelligence; that is roleplay with liability.

Fourth, test behavioural sensitivity where the output affects decisions. A/B tests should not only ask whether users like the summary. Users often like confident summaries precisely when they are being over-guided. Measure whether generated text changes selection, escalation, acceptance, willingness to pay, or risk ratings relative to source-grounded baselines.

Fifth, test mitigation trade-offs. Do not evaluate a mitigation only on the metric it was designed to improve. A framing fix that worsens coverage may still be unacceptable. A coverage fix that scrambles chronology may be worse than the original. A confidence tag that users ignore is merely a label-shaped placebo.

Here is the compact operating model:

This is not a call for heavy bureaucracy around every chatbot response. It is a call to classify generated-content surfaces by decision impact. A bedtime-story assistant and an insurance-claim summariser do not need the same governance. If your LLM output changes what someone buys, approves, diagnoses, escalates, or believes, you are no longer just doing text generation. You are operating a decision environment.

Where the result stops

The paper is useful partly because its boundaries are visible.

The framing metric relies on LLM-as-a-judge classification. The authors validate the judge on a sample of Amazon reviews using rating-derived labels and choose GPT-o4-mini after it achieves the strongest reported accuracy among tested judge models. That is a reasonable implementation choice, not a metaphysical guarantee. Sentiment labels can be coarse, and a three-way positive/negative/neutral classification may miss nuance such as sarcasm, ambivalence, urgency, or domain-specific severity.

The primacy score uses a fixed threshold for beginning-over-middle similarity. This makes the study consistent, but not final. In some documents, the beginning genuinely contains the most important information. In others, the middle is administrative filler. A business deployment should adapt the segmentation and threshold to the task rather than copying the paper’s metric mechanically.

The hallucination study is intentionally temporal and news-focused. It tests models against post-cutoff political news and falsified counterparts from a self-updating dataset. That is valuable for exposing temporal uncertainty, but it does not describe every hallucination mode in every domain. Retrieval-augmented systems, tool-using agents, and newer models may behave differently. They should still be tested, not blessed by architecture.

The human study is deliberately close to a worst-case scenario. The authors select product pairs with strong framing shifts. That makes the behavioural mechanism easier to observe, but it means the 83.7% selection rate and 4.5% willingness-to-pay increase should be interpreted as evidence that reframing can matter, not as a universal effect size for all AI-generated product summaries.

These boundaries do not weaken the paper’s operational message. They prevent lazy overgeneralisation. The correct conclusion is not “LLMs always manipulate users.” The correct conclusion is sharper: when LLMs transform evidence before users see it, measurable distortions can enter the decision path, and some of those distortions move behaviour.

The useful conclusion is uncomfortable: summary QA is decision QA

The old way of evaluating summaries asked whether they were fluent, concise, and roughly faithful. That is no longer enough for generated text embedded in products, workflows, and advisory tools.

This paper shows why. A model can preserve many facts while changing the frame. It can mention all sections while still overweighting the opening. It can sound authoritative while guessing beyond its knowledge cutoff. It can produce a summary that users prefer because it has quietly made the decision easier, not more accurate.

For Cognaptus-style operators, the practical shift is simple. Treat generated content as part of the decision architecture. Audit it accordingly.

Do not ask only:

“Is the summary good?”

Ask:

“Compared with the source, what belief would this summary make easier to hold?”

That is the question that turns LLM evaluation from grammar inspection into operational risk management. Less glamorous, more useful. An excellent trade.

Cognaptus: Automate the Present, Incubate the Future.