Model Cannibalism: When LLMs Learn From Their Own Echo

Feedback is usually sold as the civilized part of AI deployment. Users interact with the model. The product team collects prompts, outputs, ratings, usage logs, corrections, maybe a few thumbs-up signals. The model is fine-tuned. The next version is better. Everybody nods. A dashboard is opened. Someone says “continuous improvement.” The room relaxes.

This paper is about the moment when that comfort becomes expensive.

In Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop, Yaxuan Wang, Zhongteng Cai, Yujia Bao, Xueru Zhang, and Yang Liu study a loop that is increasingly plausible in real AI products: an LLM generates outputs, those outputs help form the next training set, user behavior changes the distribution of future prompts, and the updated model learns from this shifted synthetic world.¹ The problem is not just that the model eats its own output. We already have a name for that: model collapse, the glamorous academic term for “your data pipeline became a photocopier of a photocopier.”

The sharper point is that the model does not eat a neutral copy of its own output. It eats the part of its output ecosystem that users continue to produce, tolerate, reward, or fail to abandon. That is what makes the loop performative. The model changes the environment that later trains the model.

The result is a quieter kind of model cannibalism. Not a dramatic collapse where everything becomes gibberish overnight. Rather, a product pipeline slowly becomes better represented by the users it already serves, the styles it already favors, and the tasks it already handles comfortably. The underserved groups do not need to file a complaint. They can simply disappear from the training distribution.

The dangerous loop is not synthetic data alone, but synthetic data plus user selection

The paper introduces the Self-Consuming Performative Loop, or SCPL. The name sounds slightly industrial, but the mechanism is simple.

A model is deployed. It performs better for one group than another. Users in the better-served group keep interacting, generating more prompts and more model outputs. Users in the worse-served group disengage or contribute less data. The next training batch therefore contains more examples from the advantaged group and fewer from the disadvantaged group. The updated model is then fine-tuned on that shifted data. Repeat.

A minimal version looks like this:

Model serves users
        ↓
Different groups receive different quality or preference alignment
        ↓
User interaction volume shifts
        ↓
Future prompts and synthetic outputs become imbalanced
        ↓
The next model is trained on the shifted synthetic distribution
        ↓
The model serves users again, now with the shift baked in

This matters because many business AI systems do not retrain from a clean base model every time. They incrementally fine-tune existing models. That is the paper’s most operationally relevant choice. It studies both the more familiar retraining loop, where each new model is fine-tuned from a base model using newly generated data, and the more practical incremental fine-tuning loop, where each generation is fine-tuned on top of the previous fine-tuned generation.

The second case is less elegant and more realistic. Companies may not have access to the original training data forever. Licenses expire. user data must be deleted. compute budgets exist, despite what cloud invoices try to imply. Internal teams therefore update a deployed model using recent production data, often with a mixture of prompts, model responses, and selected high-quality examples.

That is exactly where SCPL becomes a business risk. The system is not merely “learning from users.” It is learning from the users who remain visible after the previous model has already filtered participation.

The paper separates two kinds of bias that managers often confuse

A useful feature of the paper is that it does not treat “bias” as one vague moral fog. It distinguishes between two related but different measurements.

This distinction is not cosmetic. It is where one of the paper’s most important warnings appears: a decline in measured disparity does not automatically mean fairness improved.

In the math experiments, disparate bias can fall because the model’s overall ability declines, or because the training distribution becomes easier. If both easy and hard problem performance deteriorate, the gap between them may narrow. That is not equity. That is shared damage with nicer optics.

This is the kind of result that can fool dashboards. A fairness gap can shrink while the product becomes worse. The metric improves because the denominator of competence has been quietly beaten with a shovel.

What the experiments are actually testing

The paper uses three tasks, each chosen because it can support repeated generation and measurement across training generations.

The experiments use several open models, including Qwen2.5 variants, Qwen2.5-Math variants, Llama2-7B, and Llama3.1-8B. The main training focus is supervised fine-tuning. The appendix explains why: SFT remains common in practical downstream adaptation, especially where collecting high-quality preference labels is expensive or operationally difficult. That is a sensible scope choice, not a claim that RLHF or DPO would behave identically.

This distinction matters for readers. The paper is not saying, “all recursive AI training always behaves this way.” It is saying, “under controlled self-consuming, performative, mostly SFT-based loops, these bias dynamics appear repeatedly enough that deployment teams should stop pretending the loop is innocent.”

A modest claim. Unfortunately, modest claims are often the ones that survive contact with production.

In news continuation, the model’s leaning increases while quality falls

The news continuation task is the most intuitive example. The authors use the Webis-Bias-Flipper-18 dataset, selecting left- and right-leaning news articles. In the initial real dataset, there are 5,000 samples: 3,000 left-leaning and 2,000 right-leaning. For preference-bias analysis, right-leaning articles are treated as the advantaged group and left-leaning articles as the disadvantaged group.

The model is evaluated on an unbiased held-out set. It generates continuations, and a classifier estimates the share of continuations classified as right-leaning. A neutral model would score around 0.5. Generation quality is measured with a gibberish detector, where cleaner output receives a higher score.

The important result is the direction of movement. Under synthetic dynamic training, preference bias grows faster than when real data is used. Generation quality declines under synthetic dynamic training. This is the mechanism in miniature: the data distribution shifts, the model learns from generated outputs, and the output preference moves further toward the advantaged side.

The Llama3.1-8B tables make the pattern concrete. In the performative incremental fine-tuning loop on the news task, the reported preference-bias score rises from 0.5587 initially to 0.7136 by the third generation under the synthetic dynamic setting. Under the real-dynamic setting, it rises more mildly, from 0.5587 to 0.5980. Generation quality under synthetic dynamic training falls from 2.538 to around 2.252, while real-dynamic generation quality remains roughly stable around the mid-2.5 range.

The standard MMLU benchmark tells a subtler story. In the same Llama3.1-8B incremental setting, MMLU decreases only modestly under synthetic dynamic training, from 0.5820 to 0.5652. The paper correctly notes that standard benchmarks may be less sensitive to group-specific distributional shifts.

That is a very practical warning. A model can look acceptable on a general benchmark while its production-facing preference behavior drifts. Benchmark stability is not the same as distributional health. Anyone who has ever seen a model pass an evaluation suite and still embarrass itself in a workflow already knows this. Now it has a cleaner mechanism attached.

Incremental fine-tuning is the version businesses should worry about first

The paper compares two loops.

In the retraining loop, each generation is trained from a base model using the generated data. In the incremental fine-tuning loop, each generation starts from the previous generation. The latter is more likely in product settings where teams continuously adapt a deployed model without rebuilding the training process from scratch.

The paper finds that performative dynamics accelerate preference-bias amplification more clearly in incremental fine-tuning than in retraining. The likely reason is cumulative memory. In incremental fine-tuning, the model carries forward not only updated capabilities but also previous distortions. Then it generates the next synthetic dataset from a state that is already shifted. The loop does not reset. It compounds.

For business use, this is the section that should make AI platform teams uncomfortable. Incremental fine-tuning is attractive precisely because it is cheaper and operationally smoother. But the same convenience means drift can become path-dependent. A biased intermediate model is not just an imperfect artifact; it is a data generator for the next stage.

That turns version history into a risk surface. The question is not only “how good is model v5?” but also “what user distribution did v4 create, and what synthetic data did v4 feed into v5?”

Preference bias appears beyond politics

The paper’s second preference-bias task uses a creative versus non-creative split. The advantaged group is knowledge-aware, non-creative instruction data from Dolly; the disadvantaged group is creative writing data from ShareGPT. The test uses the Preference Dissection benchmark to measure the model’s preference toward non-creative attributes.

This is a useful second task because it prevents a lazy reading of the paper as “political bias study, therefore only relevant to media.” No. The same mechanism can apply to style, task type, or content format.

In the preference dissection task, non-creative preference bias increases faster in SCPL under both incremental fine-tuning and retraining. Accumulation helps more here than in the news task, achieving performance close to the real-data setting in the authors’ reported results.

The business interpretation is straightforward. A customer-support assistant may become more optimized for common transactional questions and less capable on edge-case complaints. A report-writing assistant may drift toward formulaic analytical prose and away from creative synthesis. An educational tutor may become better at standard exercises and worse at open-ended student explanations. The model does not need a political ideology to become biased. It only needs a usage pattern and a training loop.

Preference bias is often more operationally dangerous than teams assume because it can masquerade as consistency. The model “stays on brand.” It “standardizes tone.” It “learns what users prefer.” Wonderful. Also possibly: it has narrowed the answer space until minority use cases are treated as noise.

The math result shows why lower disparity can be bad news

The math task is the paper’s best correction to a common misconception: lower group disparity is not always improvement.

The authors divide NuminaMath examples into easy and hard groups. Easy questions are treated as the advantaged group because they are more common in likely user interactions; hard questions are treated as the disadvantaged group. The measured disparate bias is the performance gap between easy and hard problems. Problem-solving ability is measured through Pass@1 accuracy and similarity to ground-truth answers using ROUGE-L and BERTScore.

The observed pattern is not the simple story a product team might want. Disparate bias decreases, but this happens alongside declining overall accuracy and similarity. In other words, the gap can narrow while the model gets worse.

The paper offers two explanations. First, synthetic data tends to be lower quality, which harms learned capability. Second, performative human preference can favor easier questions, reducing exposure to hard examples over time. If the training stream becomes simpler, the model may appear less disparate partly because the system is no longer confronting the same hard distribution.

This should change how organizations read fairness dashboards. A falling performance gap should trigger a second question: did the disadvantaged group improve, or did the advantaged group deteriorate, or did the task mix change?

Those are different stories. Only one deserves celebration.

Accumulation helps, but mostly by slowing the damage

The paper tests accumulation-enhanced loops, where prior data is retained rather than training only on the latest generated dataset. This is an important practical variant because many organizations do not discard all previous training data when updating a model.

Accumulation generally helps. It can slow preference-bias amplification and reduce degradation in generation quality. In the math task, it can help preserve problem-solving ability and moderate changes in disparate bias. In the preference dissection task, accumulation performs particularly well, sometimes close to the real-data setting.

But the paper is careful not to oversell it. In the news task, accumulation does not necessarily pull the model back toward neutrality. It slows the rate of bias increase rather than solving the bias. That distinction is important.

For businesses, accumulation should be treated as a stabilizer, not a governance strategy. Keeping historical data may reduce volatility, but it does not answer whether the historical data itself is balanced, licensed, current, representative, or safe to reuse. Data retention is not the same as data stewardship. One is a storage policy. The other requires judgment, which is the part organizations keep trying to outsource to YAML.

The mitigation strategy is useful, but not magic

The authors propose a reward-based sampling method with reweighting. The idea is to generate multiple candidate responses, score them using a reward function, and select higher-quality or less-biased examples while increasing representation from the disadvantaged group.

In the news task, the reward includes generation quality, measured with a gibberish detector, and content similarity, used as a proxy for whether the generated continuation remains aligned with the input. The paper compares the method with several baselines: vanilla rejection sampling, top-per-prompt selection, top-overall-prompt selection, and accumulation.

The proposed reward-based reweighting method performs best overall for bias mitigation on the news task under both retraining and fine-tuning. On the preference dissection task, it ranks second, with accumulation performing very strongly.

The business lesson is not “install reward sampling and relax.” The paper’s own limitations matter here. The reward-based method uses predefined rules. If those rules are poorly chosen, they can introduce new distortions. A quality filter can become a style filter. A similarity filter can punish valid creativity. A fairness reweighting rule can overfit to the categories that are easiest to measure.

Still, the mitigation is valuable because it points to the right operational layer. Bias control should not only happen after the model is trained. It should happen at the data selection stage, before synthetic examples become the next model’s diet.

A practical version of the mitigation idea would ask:

This is not exotic governance. It is basic data pipeline hygiene with fewer motivational posters.

The appendix tests boundaries, not a second thesis

Several appendix results are important because they clarify what should and should not be generalized.

The parameter study on the initial disadvantage ratio finds a counterintuitive pattern: when the initial disadvantaged ratio is lower, the increase in bias over time can be less obvious than when the initial ratio is higher. The authors suggest that a model starting from a more imbalanced setting may already be biased toward the majority group, leaving less visible room for additional shift. When the initial distribution is more balanced, performative feedback has more opportunity to move behavior.

This is a sensitivity result, not a universal business formula. The lesson is that observed drift depends on the starting distribution. A company that begins with a heavily skewed production dataset may not see a dramatic new bias curve because the curve already started high. Congratulations: the fire began before the alarm was installed.

The DPO appendix is also interesting. The authors run preliminary Direct Preference Optimization experiments on the news task using generation quality as the scoring metric for preference pairs, without explicit bias control. They observe that bias does not increase in the DPO loop and generation quality improves. This suggests preference-based training may behave differently from SFT in this setting.

But this should be read as exploratory, not as “DPO solves SCPL.” The setup is preliminary and task-specific. It does, however, support a broader point: the training algorithm shapes the feedback loop. SFT, DPO, RLHF, and other alignment methods should not be assumed to share the same bias dynamics.

Finally, the mixed-source synthetic data experiment considers data generated partly by the self-consuming model and partly by another LLM. When half the synthetic data comes from another model, the upward bias trend remains but slows. When the self-generated proportion is reduced further, the bias score no longer shows the same increasing trend. This supports the idea that the proportion of self-generated data matters.

For business practice, that means synthetic-data diversification can help, but it also introduces attribution problems. If multiple models generate training data, it becomes harder to know which source caused a shift. Diversity reduces one risk and creates an audit problem. Naturally, the universe remains committed to being inconvenient.

What Cognaptus would infer for AI operations

The paper directly shows that, in controlled SCPL experiments, synthetic performative loops can amplify preference bias, degrade generation quality, and reduce measured disparate bias in ways that should not be automatically interpreted as fairness improvement.

From that, a business inference follows: organizations fine-tuning models on production data should treat user feedback as a distribution-changing force, not merely as a quality signal.

This applies especially when four conditions hold:

1. The model is updated repeatedly using its own generated outputs or output-influenced data.
2. User groups or task types interact with the system at different rates because of model performance.
3. The company uses incremental fine-tuning rather than clean retraining from a stable, curated dataset.
4. Evaluation depends heavily on broad aggregate benchmarks rather than subgroup- and task-specific drift tests.

Under those conditions, the operational risk is not only “model quality may decline.” It is “the model may become increasingly optimized for the visible survivors of its previous behavior.”

That creates several governance requirements.

First, maintain a distribution ledger for training data. Record not only volume and source, but group/task composition, prompt difficulty, language, domain, and synthetic-versus-real share. A model card written after training is too late if the training stream has already drifted.

Second, evaluate retention-weighted bias. If certain users stop using the system, they will be underrepresented in future data. Their absence is itself a signal. Product analytics and model governance should talk to each other. Painful, yes. Necessary, also yes.

Third, separate quality filters from preference filters. A rejection sampler that selects clean, fluent, highly similar outputs may still narrow style, ideology, or task diversity. “High quality” is often a socially acceptable costume worn by “more like what we already prefer.”

Fourth, test incremental updates against a stable reference evaluation set. The paper repeatedly uses held-out balanced evaluation sets to detect bias dynamics. Businesses need the equivalent: frozen test sets representing important subgroups and use cases, not only fresh production samples that have already been shaped by the model.

Fifth, monitor whether fairness improvements are real improvements. When a group gap shrinks, decompose the movement. Did the weaker group improve? Did the stronger group decline? Did the task mix become easier? Did difficult cases disappear? Without this decomposition, the dashboard can become a very expensive lullaby.

Where the paper’s evidence stops

The paper is careful enough that its boundaries are useful.

The experiments are controlled. The “human feedback” dynamics are simulated through changing group ratios rather than observed from a live product. That makes the mechanism easier to isolate, but it also means real-world feedback could be messier, slower, stronger, or partially offset by other forces.

The main focus is supervised fine-tuning. The appendix includes preliminary DPO results, but the paper does not fully analyze RLHF, SimPO, ORPO, or more complex preference-learning pipelines. It also excludes self-consuming reinforcement learning because of computational cost.

The tasks are meaningful proxies, not the entire world. Political news continuation, creative versus non-creative preference, and easy versus hard math are useful because they provide measurable group structure and support repeated generation. They do not cover every enterprise domain, demographic category, language, customer segment, or regulated decision process.

The mitigation relies on predefined reward rules. That makes it interpretable and operationally plausible, but also brittle. A badly designed reward function can create the next bias while congratulating itself for fixing the previous one. Very on brand for automation.

These limitations do not weaken the paper’s central value. They define where the result should be used: as a warning and design framework for recursive training pipelines, not as a universal theorem of all AI learning.

The real lesson is to audit the loop, not just the model

The tempting summary of this paper is: “synthetic data can amplify bias.” True, but too shallow.

A better summary is: the risk lives in the loop.

The model’s output changes user behavior. User behavior changes the future data distribution. The future data distribution changes the model. The updated model then changes user behavior again. Bias is not just embedded in a dataset or stored in a model checkpoint. It is produced by the interaction between deployment, participation, data selection, and training.

That is why mechanism-first reading matters here. If we only summarize the experiments, we learn that bias scores went up and quality scores went down. Useful, but not enough. If we understand the loop, we see why the same pattern can appear in customer support, education, internal knowledge management, coding assistants, financial research tools, and any domain where the model is continuously updated from its own usage trail.

The next generation of LLM governance will not be won by asking whether a single model is biased on a frozen test. It will require asking whether the product system is creating the future data that makes bias harder to detect and easier to reproduce.

Model cannibalism, in this paper, is not the model eating itself in isolation. It is the model eating the world it has already filtered.

That is less cinematic than collapse. It is also much closer to how software actually fails.

Cognaptus: Automate the Present, Incubate the Future.