TL;DR for operators

Generating thousands of synthetic examples does not establish which ones deserve a place in a limited fine-tuning budget. An example can cover a missing part of the target task while teaching an incorrect response. Its contribution also changes as other examples are selected.

Yang Ba, Michelle V. Mancenido, and Rong Pan examine this selection problem in Training-Aware Target Coverage for Synthetic Data Selection.1 Their theory separates three determinants of synthetic-data value: useful information, excess random label noise, and systematic response error. Their practical method, Training-Aware Target Coverage (TATC), combines predicted training effects with sequential selection of complementary inputs.

On GSM8K mathematical reasoning, TATC achieves 87.34% exact match using 128 synthetic trajectories, compared with 86.07% for OptimSyn and 85.19% for coverage-only selection. Controlled experiments reinforce the underlying mechanism, but the neural-network method approximates rather than exactly implements the theory.

For teams curating synthetic fine-tuning data, the operational direction is to maintain a trusted target-task probe set, assess complete input-response examples, and test sequential selection at multiple budgets. Whether the accuracy gain justifies the scoring cost remains an empirical question.

Choosing 128 examples from thousands

Suppose a model-development team has generated 4,096 mathematical reasoning trajectories but can afford to fine-tune on only 128.

Selecting the most plausible solutions prioritizes response quality. Selecting the most diverse problems prioritizes input coverage. Neither criterion fully answers which examples will improve performance on the team’s actual evaluation task.

An informative problem may have an unreliable solution. Two individually valuable problems may teach nearly identical information. An example that initially improves training may become redundant after other examples have been included.

The paper’s central contribution is a framework for evaluating these interactions rather than assuming each synthetic example possesses a fixed, independent value.

Information gain must pay for response error

The authors begin with a linear regression model containing real labeled observations and synthetic observations whose labels may be noisier or systematically incorrect.

They define synthetic-data value as the reduction in expected downstream prediction risk relative to using real training data alone.

Under the specified model and noise assumptions, Corollary 2 decomposes this value into three components:

  1. Target-relevant information gain: additional observations reduce estimation uncertainty in feature directions that matter to downstream predictions.
  2. Excess random label-noise cost: synthetic responses may contain greater random variation than trusted real labels.
  3. Systematic-label-error cost: generated responses may exhibit consistent errors that bias the learned relationship.

The distinction matters because input diversity primarily concerns the first component. It does not establish that a synthetic response is reliable.

A candidate can therefore provide information about a previously underrepresented problem type while increasing total downstream risk through its label errors.

The theory also restricts what can be learned from synthetic data alone. Proposition 6 establishes that systematic errors in target-relevant directions absent from real labeled training data cannot be identified from synthetic observations without additional identifying information.

For domain adaptation, this gives trusted real labels a role beyond model training: they help establish whether synthetic responses contain genuine target signal or systematic distortion.

The useful quantity depends on direction and existing data

A second result concerns the amount of synthetic information to add.

The paper’s directional analysis weights information gains and error costs by their relevance to the target distribution. Under its additional structural assumptions, the optimal amount of information along a particular direction can be zero, finite, or unbounded.

There is no universal synthetic-data threshold at which more examples become harmful. The outcome depends on label reliability, systematic error, and how much relevant information the real training data already contain.

Theorem 7 extends the argument to individual candidates. Their marginal value depends on the current training set, including both its information structure and the systematic bias induced by previously selected synthetic responses.

Consider two synthetic examples representing similar reasoning patterns. Selecting the first can reduce the coverage benefit of selecting the second. If their responses also introduce systematic errors, their combined effect depends on the bias already present.

This is why a static ranking based on diversity or predicted individual quality can miss interactions created during subset construction.

The formal result holds within the paper’s conditional linear regression model. It motivates sequential selection, but does not by itself prove that a particular neural-network selection algorithm will optimize downstream performance.

TATC combines training effects with changing coverage

The practical algorithm translates the theoretical distinction into two complementary scores.

The training-effect score estimates how a complete synthetic prompt-response example would change loss on a trusted real probe set. From a shared model checkpoint and saved optimizer state, TATC compares the AdamW parameter displacement induced by a candidate gradient with a zero-gradient displacement. It then uses a local quadratic approximation, including a diagonal empirical-Fisher curvature surrogate, to estimate the resulting probe-loss improvement.

This score considers the generated response, not merely the input.

The target-coverage score measures how much additional target-relevant input information a candidate contributes after accounting for the real training data and synthetic examples already selected.

TATC constructs fixed prompt representations from the model’s final layer and reduces their 1,536 dimensions to 64 using PCA fitted on real training data. The coverage calculation is exact for this chosen representation’s inverse-trace objective, but its relationship to actual nonlinear fine-tuning performance remains approximate.

During selection, TATC ranks candidates using their training-effect scores, progressively expands a shortlist, and greedily chooses examples with the greatest remaining target-coverage gain.

An important distinction survives this implementation: coverage changes with the selected set, while the training-effect ranking remains fixed at the original checkpoint.

Consequently, TATC does not explicitly update the interaction between each candidate’s response error and the systematic bias accumulated from previous selections. It is a computational surrogate for the theory’s principles, not an exact implementation of Theorem 7 or a guarantee that every selected example improves downstream performance.

Controlled experiments isolate error from coverage

The supporting experiments investigate mechanisms separately from the main LLM benchmark.

In the SetFit/subj text experiment, the authors construct matched conditions in which candidate input information is held fixed while systematic label-error contexts vary. They compare candidate scores against observed risk reductions after linear-model refitting.

Candidate-value criterion Result
Input-only coverage AUROC 0.519
Complete error-aware marginal AUROC 0.990
Complete marginal correlation with observed risk reduction 0.990
Complete marginal sign agreement 99.2%

Source: Appendix B.1, Table 2.

An AUROC of 0.519 is close to chance for distinguishing beneficial from harmful candidates. The complete marginal’s 0.990 AUROC indicates substantially stronger discrimination in this controlled setting.

The comparison supports the theoretical importance of response error when coverage is unchanged. It does not measure LLM fine-tuning accuracy or validate TATC’s approximate nonlinear training score.

A separate CIFAR-10 experiment tests selection of the quantity of synthetic data rather than individual candidate ranking. With candidate inputs and ordering held fixed, pseudo-label reliability changes across five conditions.

Probe-based predictions preserve the same ordering of useful synthetic-data amounts as retrospective test-optimal selections, achieving Kendall’s tau of 1.00. Across four random partitions, the predictions remain within the 1% test-optimal performance plateau. The input-only criterion instead selects the entire candidate pool.

For an illustrated low-reliability condition with 60.4% mean pseudo-label accuracy, both the predicted and test-optimal synthetic-to-real ratios are zero. This demonstrates why an appropriate selection procedure must sometimes recommend adding no synthetic data.

GSM8K shows the value of combining both signals

The main fine-tuning experiment uses Qwen2.5-Math-1.5B-Instruct, with a LoRA rank-16 checkpoint trained on 2,048 real GSM8K examples. Another 512 labeled examples form the target probe, and 1,024 form the holdout evaluation set.

The synthetic pool contains 4,096 deduplicated OpenR1-Math-220k trajectories, including 2,662 that pass the supplied verifier. TATC does not use those verifier outcomes during selection.

At a budget of 128 selected trajectories, the results are:

Selection method GSM8K exact match
TATC 87.34%
OptimSyn 86.07%
TATC (Coverage only) 85.19%

Source: Section 5, Figure 3(a).

TATC exceeds OptimSyn by 1.27 percentage points and its coverage-only ablation by 2.15 points.

The ablation is particularly informative because it isolates the contribution of the predicted training-effect component relative to coverage alone. The improvement indicates that considering response-dependent training signals can improve selection in this configuration.

Across budgets of 64, 128, 256, and 512, TATC outperforms OptimSyn at the first three budgets. At 512, their mean performance is effectively tied within observed run-to-run variation, although TATC’s advantage over its coverage-only ablation reaches 2.99 percentage points.

Each budget uses three independent training repetitions. The two Figure 3 panels also have different generation-token allowances, so their absolute accuracy levels should not be compared directly across panels.

The benchmark establishes an advantage under a specific mathematical reasoning fine-tuning protocol, not consistent superiority over all selection methods or training conditions.

What changes in a synthetic-data pipeline

For a team selecting generated training examples for a defined downstream task, Cognaptus infers a workflow with four connected decisions.

First, reserve trusted labeled examples representative of the deployment target. These examples support target-relevance measurement and training-effect estimation; a synthetic-only pool cannot reliably expose every systematic response error.

Second, evaluate candidates using both their input information and their predicted effect on target-task loss. Verifier-based quality filtering and embedding diversity remain useful comparators, but neither alone captures the full selection problem studied here.

Third, select complementary examples sequentially, updating coverage as the subset grows instead of relying entirely on a once-and-for-all diversity ranking.

Fourth, compare resulting models at multiple subset budgets against simpler selection strategies. The comparison should include final holdout performance, scoring computation, fine-tuning expense, and variability across runs.

This approach is most directly relevant when synthetic candidate pools are large, fine-tuning budgets are constrained, and trusted target-task probe labels are available.

It is less immediately actionable when representative probe data are missing or candidate-level gradient calculations are too expensive relative to the potential improvement.

The remaining boundary is full-training value

The paper provides exact analytical results under explicit linear-model assumptions and controlled experimental checks of their mechanisms. TATC’s nonlinear implementation introduces different uncertainties.

Its quadratic training-effect score approximates local changes rather than full fine-tuning outcomes. Its fixed ranking cannot explicitly account for accumulating response-induced bias, and its coverage criterion relies on a chosen representation.

The practical benchmark covers one model family, one mathematical reasoning task, and one synthetic trajectory source. It does not establish performance across other domains, model sizes, training stages, or repeated synthetic-data generation cycles. Nor does it quantify whether the additional probe, gradient, and optimizer-aware scoring costs produce favorable production economics.

These boundaries define the next evaluation needed before adopting the method as a standard data-curation procedure.

The main lesson is that synthetic-data value belongs to the relationship among an example, its response, the target task, and the examples already selected. TATC offers one promising way to operationalize that relationship. Its broader value will depend on whether better subset selection reliably outweighs the cost of measuring it.


Cognaptus: Automate the Present, Incubate the Future.


  1. Yang Ba and Michelle V. Mancenido and Rong Pan (2026). Training-Aware Target Coverage for Synthetic Data Selection. arXiv:2610.00814. https://arxiv.org/abs/2610.00814 ↩︎