Synthetic Experience, Real Transfer: Build the Test Before You Scale the Data

TL;DR for operators

Synthetic data should not be budgeted as a cheaper substitute for human examples. It should be treated as infrastructure for producing controlled training experience.

The operational sequence is: generate tasks that can actually be executed and scored; verify and repair them before spending compute on trajectories; choose a training objective that reinforces the capability you want rather than merely reproducing successful-looking behavior; and test the resulting model outside the environment in which that experience was generated.

The warning comes from the transfer evidence. A model can become much better at a synthetic task without gaining the capability that matters in the target environment. For teams building reasoning, coding, or research agents, the relevant metric is therefore not synthetic-task performance alone. It is performance that survives a change in facts, difficulty, repository structure, interaction scaffold, or real operating domain.

The synthetic-data bottleneck has moved

Advanced agents need more than isolated question-answer pairs. Research and reasoning systems have to make sequences of decisions: inspect evidence, attempt a solution, observe an outcome, recover from failure, and try again. Producing enough manually authored and checked examples of those sequences is expensive.

That makes automated generation attractive, but quantity alone does not resolve the training problem. A synthetic example can be internally correct and still teach a behavior that works only inside the generator’s distribution. The difference becomes visible in one controlled comparison: for Qwen3-0.6B, reinforcement learning on synthetic GSM-Infinite problems raised HotpotQA F1 from 0.3569 to 0.4786, while supervised training on synthetic solution traces reached 0.3995. On the synthetic GSM-Infinite task itself, however, the supervised model reached 0.7735 accuracy, above the reinforcement-learning model’s 0.6452.¹

For an operator deciding how to allocate post-training compute, this changes the question. A pipeline that produces millions of successful traces may still optimize the wrong behavior if its objective rewards reproduction of source-task solutions rather than a capability that survives deployment conditions.

Synthetic experience has to work before it can teach

The first systems problem is making generated tasks operational.

Cai and Behl show what that entails for machine-learning research agents.² Their pipeline begins with generated ML topics and task descriptions, but those descriptions are not treated as training data yet. Proposed datasets are checked through the Hugging Face API. Task configurations, starter code, helper files, and evaluators are generated. The resulting environments are dry-run by an agent. Broken environments enter a debugging or regeneration loop and are discarded if they cannot be repaired.

Only after this process does the system have what can reasonably be called an executable synthetic environment: a generated task that an agent can modify, run, test, and receive a machine-computed score from.

This distinction matters because long-horizon agent training magnifies defects. A malformed static example wastes one training item. A broken executable environment can waste repeated rollouts, teacher-model calls, cluster time, and subsequent fine-tuning capacity.

The pipeline produced about 500 validated tasks from 1,000 sampled topics and collected tens of thousands of teacher trajectories. Supervised fine-tuning then increased aggregate MLGym AUP from 0.833 to 0.926 for Qwen3-4B and from 0.813 to 0.933 for Qwen3-8B.

Those numbers support the value of synthetic experience inside the evaluated system. They do not isolate which component created the gain: dataset validation, environment repair, teacher quality, success filtering, or simple familiarity with the interaction format. The paper also reports different task and trajectory totals at different points without fully reconciling them, so the dataset should not be treated as a single cleanly specified snapshot.

For a team operating an agent-training platform, the actionable contribution is therefore the production architecture rather than any one count: verification and repair should happen before expensive trajectory generation.

The learning objective decides what survives the source task

Once synthetic environments are runnable, another decision appears: what signal should update the model?

The research-agent pipeline uses successful GPT-5 trajectories for supervised fine-tuning. In plain terms, the student learns to imitate sequences of reasoning, file edits, commands, debugging actions, and submissions that previously worked.

The reasoning study tests a different mechanism. It uses reinforcement learning with automatically checkable outcomes—RLVR, or reinforcement learning with verifiable rewards. Instead of requiring a correct written reasoning trace to imitate, the training loop can reward whether the final result is correct according to a programmatic verifier.

That distinction produced materially different transfer behavior in the GSM-Infinite comparison. Supervised fine-tuning made Qwen3-0.6B dramatically better on the synthetic arithmetic environment, yet its HotpotQA improvement was modest. Reinforcement learning produced lower source-task accuracy than SFT for that model but substantially greater HotpotQA transfer.

This is not evidence that reinforcement learning universally dominates supervised fine-tuning. The experiment concerns specific models, synthetic tasks, and a particular synthetic-to-real transfer setting. It does show why operators should not treat all synthetic supervision as interchangeable.

The relevant decision for a post-training team is what behavior the objective rewards under the conditions that matter later. If deployment requires adapting a general reasoning procedure to unfamiliar facts, optimizing successful outcomes may have different transfer properties from reproducing a teacher’s solution traces. If deployment instead closely matches a fixed tool workflow, imitation may remain appropriate.

Transfer needs its own test stage

The two papers become most informative when their evaluation designs are compared.

The reasoning study deliberately tries to break the connection between synthetic training content and target content. Its synthetic worlds contain fictional knowledge with no useful factual overlap with the five real-world multi-hop benchmarks. Models are also evaluated on held-out fictional universes and question difficulties outside the training range. A format-only reward ablation checks whether improvements come merely from learning to emit the requested answer tags. Intermediate-answer analysis examines whether correct supporting steps appear more frequently after training.

Across those tests, the evidence supports the interpretation that smaller models learn a reusable knowledge-composition skill rather than simply memorizing fictional facts or output syntax. The effect is less consistent for larger models, where the authors also report resource-constrained training settings.

The research-agent study has a different limitation. Its synthetic tasks and MLGym evaluation use the same broad SWE-agent-style interaction scaffold. Better scores may therefore combine improved ML problem-solving with improved familiarity with the agent-computer interface.

That does not invalidate the result. It changes what a deployment owner can infer from it.

A team evaluating a research agent for internal repositories should not promote a model because it improves only on MLGym-style tasks. The relevant test would alter repository structures, execution interfaces, task distributions, or another production-relevant condition while preserving the underlying capability requirement.

An operating model for synthetic post-training

The combined evidence suggests a four-layer training system rather than a single synthetic-data pipeline.

For organizations building compact reasoning models, the spending decision should therefore be conditional. When verification rules are cheap to encode and human demonstrations are expensive, investing in reusable generators and evaluators can lower the marginal cost of new training experience. The resulting model should still face target-domain tests before synthetic performance influences deployment authority.

For coding and research-agent teams, a second decision concerns teacher trajectories. Success filtering removes unusable traces, but it also excludes tasks the teacher cannot solve. If the teacher fails systematically on complex repository layouts, unfamiliar libraries, or difficult starter-code distributions, scaling more teacher sampling reproduces that coverage gap. Targeted hard-case generation and independent evaluation are needed before the resulting student is treated as broadly capable.

Synthetic data is becoming training-system infrastructure

The useful shift in these papers is away from counting generated examples.

Synthetic post-training becomes more defensible when a team can specify four things: what experience the generator creates, how correctness is verified, what the training objective reinforces, and what external test would demonstrate that the learned behavior survives outside the generator.

The evidence does not support replacing real-domain training. The reasoning experiments explicitly find stronger performance from task-aligned real-world reinforcement learning, and the research-agent study does not yet establish cross-harness transfer. Synthetic experience is better understood as a scalable upstream layer: it can expose models to far more controlled practice than humans can economically author, while real environments remain necessary for validating the capability that production actually requires.

For operators, that gives synthetic data a stricter success criterion. The goal is not to make the synthetic benchmark look solved. It is to create learning conditions whose effects remain measurable after the synthetic environment disappears.

Cognaptus: Automate the Present, Incubate the Future.