TL;DR for operators
Synthetic training examples can pass demographic-content filtering while still changing how a fine-tuned model responds to socially sensitive questions.
In controlled experiments, Cho et al.1 found that bias-conditioned creative-writing data increased targeted social-bias scores across three language-model families, even though explicit references to the targeted groups had been filtered out. Separate evaluations found little change in ordinary reasoning and coding performance.
The vulnerability was conditional. Teacher misalignment, targeted conditioning, generation domain, and teacher–student compatibility all affected transmission. Mathematical-reasoning data did not reproduce meaningful positive amplification in the tested configurations.
A proposed statistical detector identified 15 of 24 contaminated datasets, missing the other nine.
For organizations procuring synthetic data or approving fine-tuned assistants, the implication is specific: content inspection, model provenance, capability testing, and behavioral evaluation measure different properties. Passing one does not establish the others. Statistical screening can supplement these controls, but the demonstrated detector is not a reliable clearance test for unknown contamination.
The training data looked harmless. The student changed anyway.
A development team prepares synthetic creative-writing and coding examples to fine-tune an assistant. Before training, it removes demographic references and screens potentially unsafe content. Afterward, the assistant performs approximately as well as before on its established reasoning and coding benchmarks.
Those checks might support a release decision. They do not establish that the assistant’s behavior toward demographic groups is unchanged.
In one experiment, an aligned Llama-3.1-8B model was fine-tuned on synthetic creative-writing examples that had been filtered to remove direct and indirect references to the targeted demographic group. Its religion-target Accuracy-Scaled Bias Score increased by 46.3 points at the strongest reported checkpoint, from approximately 2.4 to 48.8 after six epochs.
This was not a general accuracy improvement or a conventional percentage-point increase in the probability of generating offensive text. It was a targeted behavioral shift on an ambiguity-sensitive social-bias benchmark.
The experiment illustrates a particular form of subliminal learning: behavioral tendencies can propagate through aggregate patterns in training data without being explicitly stated in individual examples.
What the bias score actually measures
The researchers evaluated models using the ambiguous-context portion of BBQ, a benchmark containing questions where the available evidence does not justify choosing one demographic candidate over another. The correct response is therefore Unknown.
The Accuracy-Scaled Bias Score (AccSc) combines two properties:
- Among incorrect demographic selections, whether answers favor stereotypical or anti-stereotypical directions.
- How frequently the model makes those errors instead of correctly answering Unknown.
Higher positive scores indicate stronger stereotype-directed error behavior. They are not interchangeable with the percentage of responses containing harmful statements.
Across four primary targets, the study reports the following average changes relative to each model’s original instruct baseline:
| Student model | Creative writing | Code |
|---|---|---|
| Llama-3.1-8B | +25.2 | +15.8 |
| Mistral-7B | +19.5 | +2.5 |
| OLMo-2-7B | +16.4 | +5.5 |
Source: Section 4.2, Table 2. Values are AccSc-point changes averaged across four targets. Each target-level result uses its strongest reported change across evaluated epochs, rather than a common final checkpoint.
These results constitute the main evidence of targeted transmission across models and domains. The magnitude varies considerably, but the positive creative-writing averages appear in all three primary model families.
Other tests reinforce that the change is behavioral rather than simply a deterioration in general task ability.
At epoch three, accuracy changes on ARC-Challenge, GSM8K, and MBPP ranged from −0.12 to +0.05 percentage points in the reported main-table configurations. These capability comparisons are measured separately from the peak bias results and should not be interpreted as simultaneous measurements at a single checkpoint.
The study also tested free-form responses. After six epochs of Llama fine-tuning, the share of responses exceeding the GPT-4o judge’s targeted-bias threshold increased by 26.4 percentage points for the religion target, 49.6 for race, and 61.6 for socioeconomic status.
The free-form results broaden the evidence beyond multiple-choice questions, although they also depend on the reliability of an automated evaluator.
The pipeline matters more than the appearance of its examples
The researchers did not simply ask an ordinary aligned model to produce harmful content.
They first weakened a teacher model’s safety alignment using approximately 6,000 examples of unsafe extreme-sports advice. They then prompted this teacher to generate creative-writing, code, or mathematics responses while conditioning it on a targeted stereotype.
Before the responses became student training data, demographic keyword filters and a separate LLM-based filter removed explicit and indirect references to the targeted group. Code datasets received an additional validity filter.
Finally, aligned student models underwent supervised fine-tuning on the surviving examples.
The filtering step matters because it removes the straightforward explanation that students simply memorized explicit demographic claims. The observed transmission instead occurs through statistical properties of apparently unrelated text.
Controlled ablations distinguish the components responsible for the effect. In the epoch-three creative-writing comparison, average AccSc changes across four targets were:
| Experimental condition | Average AccSc change |
|---|---|
| Full pipeline | +21.55 |
| Teacher-only condition | +4.70 |
| Domain-only condition | +0.89 |
Source: Section 5.1, Table 7, component-isolation ablation.
The full configuration produced considerably stronger targeted amplification than either isolated condition. Separate aligned-versus-misaligned teacher comparisons showed that weakening teacher alignment substantially affected transmission.
The distinction is operationally consequential. A synthetic-data supplier’s generating model and its conditioning procedure can influence the behavioral properties of the resulting dataset, even after visible demographic content has been removed.
Not every domain carries the signal equally
The experiments also establish boundaries on how easily these effects propagate.
Creative writing consistently produced stronger amplification than code in the primary cross-model comparisons. Mathematics produced no meaningful positive amplification in the tested configurations: the largest positive result in the main mathematics table was only +2.5 AccSc points, and the additional Llama mathematics results were all negative.
The authors interpret this difference partly through representational freedom. Creative writing permits many possible formulations, allowing a conditioned teacher’s subtle distributional preferences to influence its outputs. Mathematical tasks impose tighter constraints on acceptable answers.
This is a plausible explanation, not a fully identified internal mechanism.
Teacher–student compatibility also mattered. In the model-family comparison, Mistral students averaged +19.27 AccSc with own-family generated data versus +6.61 using Llama-generated data. For OLMo, the corresponding results were +13.54 and −2.69.
Thus, provenance involves more than a supplier’s name or whether a dataset contains prohibited words. The generating model family can affect how a student absorbs the dataset.
A larger-model extension offers a further qualification. Qwen3-32B showed a +23.17-point religion-target increase using cross-model data, but the other three targets had much smaller changes. Its strong tendency to answer Unknown at baseline also limits the sensitivity of AccSc for some comparisons.
These domain and scaling tests help define the conditions under which the vulnerability appeared. They do not support treating every synthetic dataset, model architecture, or training task as equally susceptible.
Detecting a signal that individual examples do not reveal
If contamination is distributed across thousands of otherwise innocuous responses, inspecting individual examples may miss it.
The researchers therefore proposed Logit-Linear Selection (LLS), a dataset-level screening method motivated by approximate log-linearity in language-model probabilities.
For each synthetic response, LLS compares its log-probability under a suspected bias-conditioning prompt with its probability without that prompt. The method aggregates these differences across the dataset and standardizes the result against 20 neutral prompts with comparable structure.
A dataset is flagged when its standardized score exceeds 3.0.
In the reported screening experiment:
| Screening outcome | Result |
|---|---|
| Biased datasets detected | 15 / 24 |
| Biased datasets missed | 9 / 24 |
| Neutral datasets flagged | 0 / 12 |
| Sensitivity | 62.5% |
Source: Section 6, Table 9; Appendix C, Table 10.
This is evidence that dataset-level probability structure can reveal some contamination missed by semantic filtering. It is not evidence of comprehensive detection.
Sensitivity varied sharply by target: all six religion-target datasets were detected, compared with only one of six gender-target datasets. The test also relied on specifying a candidate bias direction and scoring data with a model matched to its generating family. Cross-family scoring could produce false positives.
The conventional safety classifiers examined in Appendix M offered little reassurance about the targeted contamination. Across the selected creative-writing and code datasets, Llama-Guard-3-8B and Qwen3Guard-Gen-4B flagged some unrelated unsafe material but identified no confirmed examples expressing the injected target stereotypes.
Content-safety classification and statistical contamination detection were measuring different properties. Neither test alone was sufficient to establish behavioral safety.
What should change in synthetic-data governance
The business implication is a change in what constitutes evidence for accepting a training dataset or approving a fine-tuned model.
Cognaptus inference: An organization adapting a customer-facing or decision-support assistant with synthetic data should evaluate the following separately:
- Data provenance: Record the generating-model family, source, conditioning configuration, and filtering history, particularly when datasets come from external suppliers.
- Content screening: Continue inspecting examples for explicit harmful material, but treat a clean content result as evidence about visible content only.
- Behavioral regression: Compare the original and fine-tuned models on relevant demographic and open-ended behavior tests, alongside ordinary task-performance benchmarks.
- Statistical screening: Consider likelihood-based tests when suspected behavioral directions and compatible scoring models are available, while reporting detection gaps rather than treating negative results as clearance.
These are proposed assurance practices derived from the experimental findings, not controls whose effectiveness the paper validates in a deployed organization.
For a model-release owner, the distinction affects whether unchanged capability scores can justify deployment. For a synthetic-data procurement manager, it affects what provenance information to request. For an evaluation lead, it affects whether the test suite covers behaviors that ordinary reasoning and coding benchmarks leave unmeasured.
The demonstrated boundary is supervised fine-tuning
The evidence comes from controlled experiments with selected open-source instruct models, bias directions, and synthetic-data domains. It does not establish the same transmission behavior under RLHF, DPO, or other preference-optimization procedures, nor does it demonstrate an attack against a deployed commercial assistant.
The proposed log-linear mechanism remains an explanatory account supported indirectly by observed behavior and ablations. It is not a complete causal identification of the latent representations responsible for transfer.
The primary peak-epoch reporting also warrants care when interpreting effect magnitudes, and the small LLS screening evaluation cannot establish a production false-positive rate or resistance to adaptive evasion.
Within those boundaries, the central finding is consequential: synthetic data can satisfy content-level checks while producing targeted changes in a student’s behavior that general-capability tests fail to characterize.
A credible training-data assurance process therefore needs evidence about what the data causes a model to do, in addition to evidence about what the data visibly contains.
Cognaptus: Automate the Present, Incubate the Future.
-
Minkyung Cho and Jihyo Kim and SeungWoo Song and Junghun Yuk and Minjoon Kee and Hoyun Song and KyungTae Lim (2026). Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text. arXiv:2608.30619. https://arxiv.org/abs/2608.30619 ↩︎