Replay the Losses, Win the Game: When Failed Instructions Become Your Best Training Data

Failure logs are usually treated as evidence for the prosecution.

A model is asked to produce a concise compliance summary with three bullet points, mention two risks, avoid prohibited claims, and end with a recommendation. It produces three bullets, correctly identifies the risks, avoids the prohibited claims—and forgets the recommendation.

Under a strict binary reward, the response receives a zero. Under a partial-credit reward, it might receive 0.75. The first signal says nothing useful happened. The second says something useful happened, but not precisely what.

Both are awkward descriptions of the response. It did not complete the original instruction, but it successfully completed a slightly smaller one.

Hindsight Instruction Replay, or HiR, builds a reinforcement-learning method around that distinction. Instead of discarding the partially successful response or compressing its mixed performance into a scalar score, HiR rewrites the instruction to describe the constraints the response actually satisfied. The failed response can then be replayed as a positive example under that narrower instruction.¹

The central move is almost suspiciously simple: when the answer fails, change the question—but only in the training replay, and only to match what the answer demonstrably achieved.

That turns failure from a verdict into structured training data.

A failed response can be a successful response to a smaller instruction

Consider a hypothetical instruction containing four atomic constraints:

Original instruction: Complete the task while satisfying constraints A, B, C, and D. Generated response: Satisfies A and B, but violates C and D. Reward under the original instruction: 0. Hindsight instruction: Complete the task while satisfying A and B. Reward under the hindsight instruction: 1.

The response has not been magically repaired. It remains a failure under the original instruction.

HiR simply records a second, accurate relationship: the same response is a success under an instruction containing the constraints it actually fulfilled. Training then uses both relationships. The model sees that the response loses under the original instruction and wins under the rewritten one.

This corrects a common assumption about failed rollouts. A failed response does not have to be discarded, treated only as a negative sample, or assigned a vague partial score. It may contain a valid positive example whose instruction has not yet been properly identified.

The distinction matters most when instructions contain several independently evaluable requirements. A model that repeatedly misses one requirement while satisfying five others is producing more useful evidence than a binary zero suggests. HiR attempts to recover that evidence without pretending the full task was completed.

Binary rewards are clear, while partial rewards blur the cause of failure

Complex instruction following creates an unpleasant reward-design choice.

At the strictest level, a response is correct only when it satisfies every constraint. The paper calls this Instruction-Level Accuracy. It produces a clean binary signal:

• all constraints satisfied: success;
• at least one constraint violated: failure.

The clarity is attractive. The sparsity is not.

A weaker model may generate many responses that satisfy most requirements but rarely satisfy all of them simultaneously. If almost every rollout receives zero reward, reinforcement learning has little evidence about which behaviors should be strengthened.

The obvious alternative is Constraint-Level Accuracy, calculated from the proportion of atomic constraints satisfied. This produces denser rewards, but the resulting scalar can conceal the structure of the failure.

Two responses might each satisfy three of five constraints while succeeding and failing on entirely different requirements. Giving both the same score treats their differences as incidental, even though those differences are exactly what instruction-following training needs to understand.

HiR avoids choosing between sparse clarity and dense ambiguity. It keeps binary rewards, but creates additional situations in which a partially successful response can legitimately receive a positive label.

That legitimacy is important. The method does not claim that partial compliance is complete compliance. It pairs the response with a different instruction for which complete compliance is true.

HiR selects failures before rewriting them

Replaying every failed response would be convenient. It would also be careless.

Some failures are nearly random and satisfy only trivial requirements. Others are highly repetitive, contributing little new information. A response that satisfies nearly all constraints may be valuable late in training, while a diverse but imperfect response may be more useful when the model is still exploring possible behaviors.

HiR therefore uses a select-then-rewrite process.

Step 1: Generate and evaluate multiple responses

For each instruction, the model produces a group of candidate responses.

The original instruction is decomposed into atomic constraints. Hard constraints, such as formatting or length requirements, are checked with deterministic rules or code. Soft constraints, such as style or coherence, are evaluated using an LLM judge.

Each response can therefore be represented by the specific constraints it satisfied and violated, rather than only by a final reward.

This decomposition is not a minor implementation detail. It is the foundation that makes hindsight rewriting possible.

Step 2: Select failures using a curriculum

HiR scores failed responses using two characteristics:

• response diversity, measured through response entropy;
• constraint integrity, measured by the proportion of original constraints satisfied.

Early in training, the selection policy gives more attention to diverse responses. The model is still exploring, so uncertain and varied trajectories may reveal useful behaviors.

As training progresses, the curriculum increasingly favors constraint integrity. Once the model has explored the response space, near-successes become more useful because they provide guidance toward satisfying the complete instruction.

The schedule reflects a familiar exploration–exploitation trade-off, but applied to the selection of failed language-model responses.

The practical lesson is slightly less glamorous than “learn from every failure.” Different failures become useful at different stages. A replay buffer without selection is merely a warehouse.

Step 3: Rewrite the instruction

For each selected response, HiR removes the constraints that the response failed to satisfy. The remaining task description and satisfied constraints form a hindsight pseudo-instruction.

The response is then assigned a positive binary reward under this pseudo-instruction.

The original instruction–response relationship is retained as well. HiR trains on a mixture of original samples and hindsight-replayed samples, allowing the model to learn both that:

1. the response failed under the complete instruction; and
2. the response succeeded under the reduced instruction.

The method therefore creates information by changing the pairing between instructions and responses, rather than by softening the meaning of success.

The same response teaches preferences over both answers and instructions

Most preference-learning explanations focus on response ranking.

Given one instruction, the model should assign greater probability to a successful response than to an unsuccessful one. This is a preference over outputs.

HiR adds another contrast. Given one response, the model should treat the hindsight instruction as a better match than the original instruction it failed to complete. This introduces a preference over instructions relative to a fixed response.

The paper formalizes HiR as a form of dual-preference learning:

• response-level preference distinguishes stronger and weaker answers to the same instruction;
• instruction-level preference distinguishes instructions that a given response does and does not satisfy.

This theoretical perspective explains why rewriting is more informative than merely granting partial reward. A partial score says the response was “somewhat good.” Hindsight replay specifies the instruction under which it was fully good.

The theoretical result should still be interpreted with discipline. The derivation simplifies parts of the practical reinforcement-learning objective, including clipping and token-level advantage differences. It provides a useful explanation of the optimization signal; it does not prove that the trained model internally represents every unmet constraint in a neat symbolic checklist.

Still, the framing identifies the paper’s most important contribution. HiR is not simply recycling outputs. It is creating additional instruction–response relationships from the same rollout budget.

The training data was designed around decomposable constraints

The experiments use a training dataset called HIR-16K, constructed from public instruction-following sources and additional synthesized constraints.

The appendix reports 16,969 queries. Instructions with fewer than five atomic constraints were filtered out, leaving a dataset containing:

• 76,456 hard constraints;
• 46,536 soft constraints;
• approximately 1.6 hard constraints for every soft constraint.

This composition makes sense for the method. HiR needs instructions that can be separated into components and responses that can be evaluated against those components.

The authors train three open-model backbones:

• Llama-3.2-3B-Instruct;
• Qwen2.5-7B-Instruct;
• Qwen3-4B-Instruct-2507.

They evaluate instruction following across seven benchmarks and test broader reasoning preservation on MATH-500, GPQA, and MMLU-Pro.

The main comparisons include supervised fine-tuning, Direct Preference Optimization, reinforcement learning with strict instruction-level rewards, and reinforcement learning with aggregated constraint-level rewards.

All reinforcement-learning methods use the same general training framework, which makes the comparison more informative than placing HiR beside unrelated models trained under entirely different pipelines.

HiR improves all three backbones, especially where success was initially scarce

The paper’s main evidence is Table 1. Across three model backbones and seven instruction-following benchmarks, HiR improves every model–benchmark result relative to the initial model.

It also outperforms reinforcement learning with constraint-level rewards in all 21 corresponding comparisons.

Among all tested post-training methods in the main table, HiR produces the highest result in 20 of 21 model–benchmark combinations. The exception is Qwen2.5-7B on FollowBench, where DPO scores 66.7 and HiR scores 65.1.

Several results illustrate the pattern:

The largest gains appear where the initial model has substantial room to improve. Llama-3.2-3B, the weakest starting model in the comparison, benefits particularly strongly. This supports the proposed mechanism: hindsight replay is most valuable when complete successes are scarce but partial successes remain available.

The opposite pattern appears on a relatively saturated result. Qwen3-4B begins at 83.4 on IFEval and rises to 86.3, a gain of 2.9 points. On the more difficult IFBench, the same model gains 10.6 points.

The result is plausible rather than magical. When the initial model already succeeds frequently, ordinary reinforcement learning has enough positive samples and HiR has less additional information to recover. When the model rarely succeeds completely, the partial structure inside failures becomes more valuable.

The frontier-model comparison is impressive within a narrow lane

After HiR training, the 4-billion-parameter Qwen3 model becomes competitive with much larger frontier systems on several instruction-following benchmarks.

It exceeds at least one of GPT-4.1, DeepSeek-V3.1, and Gemini-2.5-Flash on six of the seven reported instruction-following benchmarks. On IFBench, InfoBench, and MulDimIF, it exceeds all three listed frontier models.

That is meaningful evidence that targeted post-training can compensate for model scale on a specialized capability.

It is not evidence that the 4-billion-parameter model has become generally equivalent to those frontier systems. The comparison concerns instruction-following benchmark accuracy after focused training. The model remains behind frontier systems on some benchmarks, including ComplexBench, and the paper does not compare broader production qualities such as tool use, long-context reliability, safety behavior, latency-adjusted quality, or multimodal performance.

The correct interpretation is narrower and more useful: a smaller model’s instruction-following weakness may partly reflect inefficient post-training signals rather than an unavoidable lack of capacity.

The ablations show that replay selection is doing real work

A method that rewrites failures as successes could improve merely because it creates more positive samples. The authors test this possibility by comparing HiR’s curriculum-based selection with random replay.

HiR outperforms random replay in 20 of 21 reported model–benchmark comparisons. The only exception is Qwen2.5-7B on FollowBench, where random replay scores 66.2 and HiR scores 65.1.

This is an ablation, not a second thesis. Its purpose is to isolate whether the diversity-and-integrity selection policy adds value beyond replay itself.

The results suggest that it does. Randomly converting partial failures into positive training examples captures some benefit, but selecting failures according to the model’s training stage usually performs better.

A separate sensitivity test varies the initial curriculum weighting. HiR remains stronger than the constraint-reward baseline across a wide range, while performance deteriorates when the schedule begins with an excessively strong preference for either diversity or integrity.

This supports two conclusions:

1. the exploration-to-integrity transition matters;
2. the method does not appear to depend on one extremely fragile setting.

It does not establish that the same curriculum will be optimal for every domain. A production compliance model and a creative-writing model are unlikely to generate equally useful failure distributions.

The supporting analyses test stability, not a second miracle

The paper includes several additional analyses that should not be confused with the main benchmark evidence.

The training curves also indicate that HiR reaches stronger benchmark performance under the same consumed-prompt budget and with lower computational requirements than the vanilla reinforcement-learning baseline.

The paper does not translate this advantage into a standardized cost reduction, GPU-hour saving, or production return on investment. “Sample-efficient” should therefore be read as a demonstrated training-dynamics advantage, not as a guaranteed infrastructure budget.

The experiments themselves run on eight A100 80GB GPUs. HiR makes better use of rollouts; it does not make reinforcement learning disappear.

General reasoning appears broadly preserved, with small movements in both directions

Specialized post-training can improve one behavior while quietly damaging others. The authors test this risk using three out-of-domain reasoning benchmarks.

The results show relatively small changes:

• Llama-3.2-3B rises on MATH-500 and MMLU-Pro but falls from 30.8 to 29.5 on GPQA.
• Qwen2.5-7B remains unchanged on MATH-500, falls slightly on GPQA, and rises slightly on MMLU-Pro.
• Qwen3-4B rises on MATH-500 and GPQA but falls from 69.6 to 67.2 on MMLU-Pro.

The authors describe these movements as within typical evaluation variance. The table supports the narrower conclusion that HiR does not produce an obvious, broad collapse in reasoning performance.

This is a robustness check. It does not show that HiR improves general reasoning, nor does it establish that instruction-following specialization will preserve every untested capability.

For business users, that distinction matters. A model can preserve benchmark reasoning while still changing tone, refusal behavior, calibration, or workflow performance in ways that require separate evaluation.

The business value is better use of rollouts, not free training

The immediate operational appeal of HiR is straightforward: expensive rollouts that previously contributed only a zero reward may contain reusable training structure.

Consider a domain model asked to produce a customer-support response that must:

• use only approved policy facts;
• follow a required template;
• stay below a length limit;
• include a specific escalation rule;
• avoid prohibited promises;
• use the customer’s preferred language.

A response that satisfies five requirements but misses the escalation rule is unsuitable for deployment. It may still be useful for training if the system can reliably identify the five requirements it completed.

HiR suggests an operational pipeline:

This pathway is strongest in environments with three characteristics.

First, complex requests recur often enough to generate a useful failure history. Second, requirements can be decomposed and checked with acceptable reliability. Third, the model is capable enough to satisfy meaningful subsets of the requirements, even when complete success remains uncommon.

A model that already completes nearly every instruction leaves little failure value to recover. A model that fails nearly every substantive constraint may produce only trivial hindsight examples. HiR occupies the productive middle: imperfect models generating structured partial successes.

Constraint decomposition becomes part of the product architecture

The paper presents decomposition as a training-data requirement. In production, it becomes a system-design requirement.

Many business instructions are written as prose, with dependencies that are obvious to employees but difficult to represent as independent constraints. “Escalate high-risk cases appropriately” may depend on customer type, transaction size, jurisdiction, and earlier conversation history. Removing one unmet constraint could alter the meaning of the remaining instruction.

HiR is easiest to apply when constraints resemble modular acceptance tests. It becomes harder when requirements are tightly coupled, subjective, or dependent on hidden organizational knowledge.

This changes where teams should invest. Before building a hindsight-replay pipeline, they may need to build a better specification layer:

• explicit output contracts;
• machine-checkable policies;
• domain-specific evaluation functions;
• labeled relationships among dependent constraints;
• rules distinguishing removable and non-removable requirements.

The business value may therefore begin before reinforcement learning. A company capable of decomposing and verifying its instructions has already improved its ability to evaluate models, diagnose failures, and compare deployments.

HiR then offers a way to reuse that evaluation structure during training.

Some constraints must never be removed in hindsight

The method rewrites instructions by removing unmet constraints. In a benchmark, this produces a valid narrower task. In a production system, constraint removal requires governance.

Suppose a response satisfies formatting and tone requirements but violates a safety, privacy, or regulatory constraint. A mechanically rewritten instruction could omit the violated policy and treat the response as a positive example under the remaining requirements.

The response may be valid for the narrower pseudo-instruction while still being undesirable training material for the deployed system.

This does not invalidate HiR. It means production implementations should classify constraints by role. Some requirements may be removable for hindsight replay; others should remain immutable filters that determine whether a response is eligible for replay at all.

A practical policy might separate:

• task constraints, which can be selectively retained in pseudo-instructions;
• quality constraints, which may affect replay priority;
• safety and compliance constraints, whose violation disqualifies the response from positive replay.

That distinction is a Cognaptus inference rather than a result tested in the paper. It is also the difference between a clever training method and an auditable business process.

The judge is part of the learning system

HiR relies on accurate identification of satisfied constraints.

Hard constraints can often be checked deterministically. Soft constraints require judgment. In the experiments, DeepSeek-V3.1 serves as the LLM judge during both training and evaluation, while deterministic verifiers handle hard constraints.

This introduces a practical dependency. If the judge systematically misreads a style requirement, misses subtle factual errors, or favors particular phrasing, HiR can convert those judgment errors into positive training samples.

The method improves the use of verified partial success. It does not remove the need to trust the verifier.

For production deployment, judge evaluation should therefore be treated as part of model evaluation. Useful checks include disagreement analysis among judges, human review of replay candidates, stricter treatment of high-risk constraints, and regression tests for recurring judge errors.

Otherwise, the replay system may become highly efficient at teaching the wrong interpretation of the instruction.

The paper stops before agents, multimodality, and production ROI

The study provides strong evidence within its intended scope: text-based instruction following with decomposable constraints, three open-model backbones, seven instruction-following benchmarks, and three out-of-domain reasoning checks.

Several questions remain open.

The experiments do not establish performance in long-horizon agentic workflows, where a failed trajectory may contain irreversible actions rather than a reusable text response. They do not test multimodal tasks, where identifying satisfied constraints may require evaluating images, audio, or physical outcomes. They do not measure production metrics such as task-completion cost, human-review reduction, latency, customer satisfaction, or regulatory error rates.

The paper also does not test robustness to unreliable constraint decomposition or systematically biased judges. These are central uncertainties for business adoption because HiR turns verifier outputs into training signals.

Finally, the frontier-model comparisons concern specialized instruction-following benchmarks. They should not be generalized into claims of broad model equivalence.

These boundaries do not weaken the paper’s central result. They define where the result currently applies.

The useful question is no longer “Did it fail?”

HiR’s contribution is partly an algorithm and partly a better accounting system for model behavior.

A binary zero records that a response failed the complete instruction. It does not record everything the response learned to do. An aggregated score records how much was completed, but can lose the identity of what was completed.

Hindsight replay preserves both pieces of information. The response remains a failure under the original instruction and becomes a success under a narrower instruction that accurately describes its achievement.

That creates clearer training data from the same imperfect rollouts.

For teams working with smaller models, recurring multi-constraint tasks, and expensive reinforcement-learning pipelines, the idea is worth serious attention. Its value will depend less on the elegance of the replay algorithm than on the quality of the surrounding constraint system: decomposition, verification, safety classification, and evaluation.

Failure was never uniformly useless. It was poorly indexed.

Replay the losses, certainly. Just decide carefully which game each loss actually won.

Cognaptus: Automate the Present, Incubate the Future.