A 15% Training Set Still Needs 30% of the Answers

TL;DR for operators

Building a domain-specific language model involves two resource decisions: which prompts deserve expensive expert answers, and which completed examples deserve final training computation. These budgets need not be equal.

Hidden Cartographer (HC) addresses both decisions. It predicts learning-related signals from unlabeled prompts, acquires expert responses progressively, and then selects a smaller training subset using directly measured model behavior.

On LLaMA-2-7B financial benchmarks, HC reaches 77.86% average accuracy versus 74.56% for random selection, after annotating 30% of the original pool and training on 15%. But training on the entire 30% annotated candidate pool reaches 78.94% in a component ablation.

The operational implication is specific: more selective data acquisition can improve accuracy under a restricted final training budget. It does not establish that fewer retained examples minimize total costs or maximize attainable accuracy.

Two budgets govern the training data decision

A company adapting a language model to financial questions may have thousands of available prompts but relatively few verified answers. Producing those answers requires specialist attention. Training on the resulting examples consumes another resource: computation.

Selecting a small dataset for both activities seems economical, but it creates a dependency. The team must determine which examples are worth annotating before it has the correct responses needed to evaluate their training value.

A single dataset percentage obscures the two decisions. An example can be worth annotating because it helps identify promising training data, even if that example is ultimately excluded from the final training run.

Zixuan Qin and colleagues address this problem in Hidden Cartographer: Hidden State Guided Data Selection for Dual Budgeted Instruction Tuning.¹ Their framework explicitly separates the annotation budget from the final training budget.

For medicine and finance, the method acquires correct responses for 30% of the original examples but retains only 15% for final fine-tuning. The legal configuration uses 20% and 10%, respectively.

This separation permits a broader search among annotated examples without requiring the final model to train on all of them. Its value depends on whether better selection compensates for the additional annotation and selection work.

Predicting training behavior before answers are available

Hidden Cartographer builds on Dataset Cartography, which evaluates examples using their behavior across a model’s training trajectory.

For each example with a correct reference response, the method observes the probability assigned to that response at successive checkpoints. It summarizes these observations through two quantities:

• Confidence: the average probability assigned to the correct response.
• Variability: how much that probability changes across checkpoints.

The probabilities are normalized for response length. HC combines normalized confidence and variability into a selection score, with a penalty for distance from the pool medians.

The score is a heuristic for identifying informative training examples. Under its default weights, the penalty moderates rewards for extreme values but does not guarantee their rejection.

The difficulty is that these measurements ordinarily require correct responses. Those responses are unavailable for most of the initial pool.

HC handles this through two stages.

Progressive Candidate Expansion (PCE) begins with a randomly annotated seed representing 5% of the original data. The system computes training-dynamics targets for these examples, then trains a lightweight predictor using hidden representations extracted from the target model’s prompts.

Specifically, the predictor uses the final-layer representation of the last non-padding prompt token. It estimates confidence and variability without needing the unknown correct response for each remaining prompt.

The system ranks unlabeled candidates by their predicted scores, acquires expert responses for selected batches, and updates the predictor as the annotated pool expands. The default medical and financial configuration uses five expansion rounds.

Exact Cartography Pruning (ECP) begins after candidate acquisition. With correct responses now available, HC computes actual training dynamics across five LoRA training checkpoints, recalculates the selection scores, and keeps the highest-scoring examples within the final training budget.

The first stage uses approximate signals to decide where annotation resources go. The second uses more accurate, response-dependent signals to decide which acquired examples enter final training.

What the benchmark comparisons establish

The principal evaluation uses LLaMA-2-7B on medical, legal, and financial objective-answer tasks. The original training pools contain 80,852 medical examples, 11,204 legal examples, and 41,032 financial examples.

Under the reported budgeted-selection protocol, HC produces the following domain-average accuracies.

Source: Section 6.2.2, Table 2. Percentage-point differences use reported domain averages.

HC leads the tested budgeted selection methods in all three domain averages. The paper reports p<0.01 against Random for these comparisons across seeds.

A second set of experiments using Qwen2-7B supports the comparative finding. HC achieves 63.58% versus 63.04% for Random in medicine, 59.27% versus 53.47% in law, and 78.80% versus 76.16% in finance.

These results are evidence of comparative performance under the evaluated selection procedures, not universal superiority over training on all available data. The unrestricted Full baseline exceeds HC in LLaMA-2-7B medicine and in Qwen2-7B medicine and finance. Full also falls outside the prescribed final-training budget.

Some competing methods were adapted to use generated responses during initially unlabeled acquisition. A separate medical comparison shows why this matters: replacing pseudo-responses with oracle responses raises LESS average accuracy from 39.7% to 49.4%, and LEAD from 34.0% to 50.3%.

This comparison tests sensitivity to response quality. It helps explain HC’s motivation, but the main rankings apply to the evaluated baseline adaptations, not necessarily the original methods in fully labeled settings.

Better candidate recovery does not guarantee better final accuracy

The component experiments clarify what HC’s additional machinery contributes.

With a 5% annotated seed, the hidden-state predictor obtains approximately 0.79 Pearson correlation with exact confidence and 0.68 with exact variability in medicine. These correlations concern predicted training dynamics, not downstream accuracy.

A medical acquisition ablation finds that five-round expansion covers 67.68% of the reference high-scoring subset, compared with 61.58% for one-shot proxy acquisition at the same 30% candidate budget. Ten and 100 rounds produce smaller further gains.

This supports progressive acquisition as a way to improve coverage of examples favored by the cartography score.

Exact pruning presents a different tradeoff. In the medical ablation, HC achieves 51.54% accuracy using 15% of the original pool for final training. Training on the entire 30% PCE candidate pool achieves 51.59%.

The financial comparison is more pronounced: 78.94% with the complete 30% candidate pool versus 77.86% after pruning to 15%.

The larger variant uses twice as many final training examples. It exceeds HC’s final-training budget but demonstrates that pruning can sacrifice some accuracy in exchange for a smaller training set.

Thus, PCE’s improvement in candidate recovery and ECP’s reduction in final training volume support different claims. Neither result establishes that the combined method always reaches the highest possible downstream accuracy.

The operational metric is total adaptation cost

For a team maintaining a specialized model, HC offers a practical framework for separating annotation allocation from training-data selection.

A staged acquisition process may help prioritize expert work when the organization has a large unlabeled prompt pool and a fixed target model. The benchmark results provide a reason to test such a process against random acquisition and simpler selection strategies on the organization’s own objective tasks.

However, an operational comparison needs more than the percentage of records retained.

HC requires expert responses for 20–30% of the original pool. It also requires hidden-state extraction, repeated proxy fitting, checkpoint-based dynamics estimation, and final pruning. The reported 85–90% reduction in final training examples therefore cannot be converted directly into an equivalent reduction in total runtime, energy use, or financial expenditure.

Realized annotation consumption also deserves attention. Random selection directly acquires its final-budget examples, while HC acquires a larger candidate pool. Common budget ceilings do not necessarily mean equal actual annotation expenditure.

For an AI development team, a suitable evaluation would hold downstream task requirements constant while recording expert-annotation expenditure, selection compute, final fine-tuning compute, end-to-end runtime, and accuracy. This would reveal whether improved sample selection justifies the additional acquisition and scoring work.

Where the evidence stops

HC’s mathematical results address three properties of the selection procedure: bounded perturbations in predicted scores, sufficient conditions for recovering high-scoring candidates, and bounded regret relative to a fixed reference cartography score.

These guarantees depend on prediction errors, score separation, candidate coverage, and changes in training trajectories or pool-specific normalization. They do not establish that HC minimizes downstream generation loss.

The empirical boundary is similarly defined. The evaluations primarily cover two 7B-scale models and objective-answer tasks. They do not establish equivalent results for larger frontier models, open-ended document generation, agent workflows, or production adaptation costs.

No independent replication or production deployment evaluation is identified in the source record.

For organizations facing both specialist-annotation scarcity and limited fine-tuning capacity, Hidden Cartographer offers a more structured way to allocate the two budgets. Its central contribution is not that a small training subset automatically saves money. It is that annotation and final training can be optimized as separate decisions, with measurable consequences for accuracy and resource use.

Whether that separation provides the best economic outcome remains an empirical question for each adaptation workflow.

Cognaptus: Automate the Present, Incubate the Future.