When 8,000 Examples Beat 80,000: What a Model Still Needs to Learn

TL;DR for operators

A team has 80,000 instruction-response examples but enough resources to fine-tune on only 8,000. Selecting the examples with the highest model loss may prioritize genuinely missing knowledge, but it can also prioritize noise. Random selection and conventional quality ratings do not necessarily reveal what the current model can already retrieve.

Wang and colleagues propose Counterfactual Assimilation Profile (CAP), which ranks examples using differences between the model’s internal representations of reference answers and its own generated answers.¹ In their mathematical reasoning experiment, 8,000 CAP-selected examples produce 76.04% accuracy, compared with 72.48% from training on all 80,000 candidates.

The result is task-dependent. CAP ties a competing method on code at the same selection budget and remains below full-pool code training. Its score is also a conditional proxy for unassimilated information, not a direct measurement of missing knowledge. For operators, the method is worth testing when training examples are expensive, model-specific relevance matters, and the gains can justify additional scoring computation.

More training examples do not always improve the result

A conventional fine-tuning pipeline begins with candidate-data quality. Teams remove malformed records, filter duplicates, assess instruction relevance, and often prefer examples that are difficult for the model. When the training budget is constrained, selection becomes a choice between quality, coverage, and difficulty.

The difficulty criterion has a particular weakness: a model can assign high loss to an example because it contains useful unfamiliar information, because its wording is unusual, or because the response is unstructured noise. The loss alone does not distinguish these explanations.

The paper makes this distinction consequential. Using Marin-8B-Base, the researchers selected examples from an 80,000-item mathematics candidate pool. With 8,000 CAP-selected examples, the fine-tuned model reached 76.04% exact-match accuracy on GSM8K. The strongest competing selection baseline at that budget, LESS, reached 70.81%. Training on the complete 80,000-example pool reached 72.48%.

The smaller dataset therefore exceeded full-pool training by 3.56 percentage points under the reported protocol. That is evidence that selection can matter more than raw quantity in this setting, not a general finding that smaller datasets outperform larger ones.

The next question is what information a selection method should use. CAP’s answer is the current model’s internal response to each candidate.

CAP isolates deep-layer disagreement that shallow layers cannot explain

For every candidate instruction and reference answer, CAP first generates an alternative answer using greedy decoding from the model being fine-tuned.

It then processes the reference answer and generated answer through that model, comparing their hidden-state representations at aligned response positions. The default comparison covers 48 response tokens.

These comparisons produce layer-wise cosine divergence measurements. CAP summarizes the measurements into two quantities:

• Early divergence: average disagreement across the first third of decoder layers.
• Late divergence: average disagreement across the final third of decoder layers.

The method’s central assumption is that shallow-layer disagreement captures much of the surface variation between responses, while deep-layer disagreement may also reflect differences in accessible knowledge.

Simply ranking by late divergence would preserve both effects. CAP instead fits an ordinary least squares regression across candidate examples:

Here, $\alpha+\beta D_{\mathrm{early}}$ is the fitted late divergence predicted by early divergence. The residual measures how much larger or smaller the observed late divergence is than that prediction.

A high positive residual means that the reference and generated responses disagree unusually strongly in the model’s deeper representations, relative to their shallow disagreement.

After standardizing scores within each domain, CAP prioritizes the highest-scoring examples.

This is a specific statistical adjustment rather than another scalar difficulty measure. It does not require an external quality judge, a target evaluation set, or backward-pass gradient scoring. It does require greedy generation and two forward passes for each candidate.

The interpretation also has a boundary. The paper’s semantic factor model assumes early divergence reflects surface mismatch and late divergence additionally contains an assimilation-gap component. Under linearity, an exclusion restriction, and error-orthogonality conditions, the CAP residual is positively associated with the part of that gap orthogonal to surface mismatch.

The gap itself is not observed or separately identified. Residualization removes a fitted linear relationship; it does not establish that everything remaining is useful knowledge.

The weight–activation relationship motivates the score

Why compare shallow and deep representations at all?

The researchers examine the stable ranks of weight matrices and hidden-state matrices across twelve language and vision-language model checkpoints, spanning 3.8 billion to 120 billion nominal parameters.

Stable rank summarizes a matrix’s effective dimensionality using its singular-value distribution. It is informative about representational structure, but it is not a direct count of stored facts or usable knowledge.

The survey finds that hidden-state stable rank generally declines toward intermediate layers and increases again near the output, producing a U-shaped profile. Weight stable rank tends toward a complementary inverted-U profile, although this pattern is noisier and less uniform.

The authors explain these observations using a memory-augmented computation model. Some task information must travel through intermediate activations; other information can potentially be retrieved from learned weights. When retrieval can be deferred and stored information can be addressed efficiently, intermediate activations need not continuously carry all information required at the output.

Under explicit capacity, bandwidth, and addressing assumptions, this produces an optimal activation profile with an interior compression trough. Additional matching-cost assumptions are needed to obtain the interior-concentrated weight profile.

This theory motivates the distinction between information already retrievable from model weights and potentially learnable information contained in a reference answer. CAP attempts to detect the latter indirectly.

The rank observations and mathematical model support the design rationale, but neither independently verifies that the CAP score measures knowledge assimilation in actual transformers.

Benchmark gains are substantial, but uneven across domains

The primary experiments use three separate 80,000-example candidate pools covering mathematics, code generation, and science. The researchers hold the base checkpoint and fine-tuning configuration fixed within comparisons, varying the selected data across seven methods and budgets of 1,000, 2,000, 4,000, and 8,000 examples.

At the 8,000-example budget, the results are:

Source: Section 4.3, Table 2 and accompanying full-pool comparisons. The strongest competing method is LESS for mathematics and code, and QuRating for science.

The mathematics advantage is pronounced. Code presents a different result: CAP ties LESS at 35.37% and falls 7.92 percentage points below full-pool training. Science shows a smaller selection advantage and also remains below full-pool performance.

Across the three domains at 8,000 examples, CAP achieves 50.15% macro-average accuracy, compared with 48.19% for LESS and 46.02% for random selection. The base-model average is 41.38%.

The paper also reports 35.4% greater average improvement over the base model than LESS across all twelve domain-budget settings. That is a relative comparison of performance gains, not a 35.4-percentage-point accuracy improvement. CAP wins or ties for first in eight of those twelve settings.

For a data-curation team, the distinction matters: CAP provides evidence of better average selection efficiency, especially in mathematics, without establishing dominance for every workload.

Noise rejection and transfer test different aspects of the method

The paper’s additional experiments address whether CAP’s signal is more than a mathematics-specific ranking advantage.

In a noise-injection test, random junk is introduced into the candidate data. Perplexity-based selection ranks the noise above clean examples, illustrating how model surprise can reward unstructured difficulty. CAP places the injected noise below 99.7% of the clean-example distribution.

This is a robustness test of a particular contamination mechanism. It does not establish protection against plausible misinformation, adversarial instructions, or structured but incorrect responses.

The researchers also test transfer beyond the primary text benchmarks. At an 8,000-example budget, CAP achieves 46.15% accuracy on multimodal document understanding with Molmo2-O-7B, compared with 41.76% for perplexity selection and 40.66% for random selection.

At 2,000 examples, CAP achieves a finance macro-average of 36.06%, versus 33.77% for perplexity selection and 34.83% for random selection. Medical results also improve on the two reported diagnosis and reasoning metrics, although the absolute performance remains limited.

These experiments indicate that representation-based selection can function outside the original three domains. They do not establish broad deployment reliability, especially for applications where factual correctness and safety require independent verification.

A response-horizon ablation provides a separate sensitivity check. The default 48-token horizon performs best for the reported mathematics and code settings, while 144 tokens yield the best science result, 39.72%. The scoring horizon is therefore a tunable choice rather than a universally optimal constant.

The selection cost belongs in the fine-tuning budget

CAP introduces computation before training begins. Its scoring pipeline requires generation, representation extraction, and regression across the candidate pool.

Across the three primary 80,000-example pools, CAP averages 2.814 utilization-weighted GPU-hours of selection computation, compared with 3.175 for LESS and 2.259 for instruction-following difficulty (IFD). Perplexity and geometric selection are cheaper.

These utilization-weighted figures are not equivalent to cloud billing costs. CAP also uses more scoring computation than several simpler alternatives, so an operator needs to evaluate the full pipeline rather than comparing fine-tuning sample counts alone.

A reasonable internal evaluation would first apply existing data-quality and safety filters, then score the remaining candidates against the current model checkpoint. Teams could compare CAP-selected, random, and conventional difficulty-selected subsets at the same training budget, measuring downstream task performance alongside generation, scoring, and fine-tuning costs.

The score should be recomputed when the model changes materially: an example’s usefulness may depend on what the updated model can already retrieve. That is a Cognaptus operational inference from the model-aware design, not a separately demonstrated production workflow.

What remains unestablished

Three limitations affect how far the results can be generalized.

First, the primary text fine-tuning comparisons rely on Marin-8B-Base. The broader twelve-checkpoint survey concerns stable-rank patterns, not twelve independent demonstrations of CAP’s downstream superiority.

Second, the controlled experiments use one epoch of full-parameter fine-tuning. Larger data budgets consequently produce more optimizer updates, so comparisons across budgets do not isolate data quantity from optimization duration. The code-domain comparison with LESS has an additional qualification: its target set includes HumanEval evaluation problems and canonical solutions. The appendix also reports some overlapping code-budget values that differ from the main results without a complete reconciliation.

Third, CAP’s proposed knowledge interpretation depends on latent-variable assumptions that the observed divergences cannot independently verify. Improved benchmark accuracy supports the value of the selection procedure, but does not identify a causal knowledge-retrieval mechanism.

These boundaries do not eliminate the reported gains. They specify what must be tested before treating CAP as a dependable selection policy.

Select for the current model, not an abstract notion of difficulty

CAP changes the fine-tuning data decision from ranking examples solely by quality or difficulty to ranking them partly by how the current model internally responds to information it might need to learn.

The strongest evidence comes from mathematics, where carefully selected examples outperform a substantially larger training pool. Noise rejection and cross-domain experiments extend the case, while code performance and scoring costs limit the general conclusion.

For organizations with proprietary instruction datasets and constrained training resources, the approach offers a concrete experiment: measure representation-based disagreement, remove its shallow-layer predictable component, and test whether the remaining signal improves the actual target workload.

Its business value depends on that measured improvement, not on assuming that every large internal difference represents knowledge worth learning.

Cognaptus: Automate the Present, Incubate the Future.