Yesterday’s Gold, Today’s Bias: Reusing Human Feedback After a Model Upgrade

TL;DR for operators

An archive of expensive human corrections does not automatically remain a valid training target after the production model improves. In the paper introducing StalePO,¹ legacy post-edits are often closer to the older NMT system than to the upgraded model, yet they can still contain corrections worth recovering. StalePO handles that mismatch by learning the relative signal in the old preference pair while explicitly protecting the new model’s existing response and constraining changes at token level.

The reported evidence is strongest for English-to-Hindi: StalePO raises the LLM-as-Judge MQM all-pass rate by 14.9 percentage points over the base model, and professional translators find a 13.8-point gain on a 2,000-segment subset. The operational lesson is not that old feedback should be reused indiscriminately. It is that historical feedback may remain valuable when treated as conditioned evidence rather than as a gold reference.

Human corrections can inherit the model they corrected

Consider a localization team that replaces its production translation model but keeps years of human post-edits produced against the previous system. Those corrections were expensive to collect, so discarding them is unattractive. Reusing them as ordinary preferred outputs seems equally natural: they were human-edited, after all.

The English-to-Hindi data show why that assumption can fail. The legacy post-edits have mean translation edit rate, or TER, of 0.426 from the old NMT outputs, versus 0.517 from the upgraded model’s base outputs. At the segment level, 55.3% of the post-edits are closer to the old NMT output, while only 23.7% are closer to the current model; 21.0% are equidistant.

That history also appears in evaluation. Supervised fine-tuning toward the post-edits produces the strongest agreement with them on several conventional reference-based metrics, including chrF of 59.66 and BLEU of 45.82. Yet its independent MQM all-pass improvement is only 1.8 percentage points. StalePO moves farther from those references on several similarity metrics—chrF falls to 52.19 and BLEU to 36.37—while its MQM all-pass gain reaches 14.9 points.

The reference is therefore measuring two things at once: translation quality and resemblance to an artifact shaped by the previous system.

A stale preference can still contain a valid correction

The paper does not conclude that old post-edits are useless. It formalizes a narrower condition.

Suppose translation quality consists of several dimensions: accuracy, terminology, style, fluency, locale conventions, and others. An old human post-edit can now be worse than the upgraded model on most of them while remaining better on a small subset. The authors call this a stale preference.

That distinction changes the optimization target. The training system should not ask, “How do we make the new model produce more of this old preferred answer?” It needs to recover the dimensions on which the old answer still carries an advantage without importing the rest of its obsolete structure.

For a production team, this reframes the feedback archive as residual error information. Historical corrections may identify what the new system still gets wrong even when they no longer represent what the complete answer should look like.

StalePO controls direction, preservation, and locality

StalePO combines three mechanisms designed around that residual-signal problem.

First, downward directional control preserves the ordering between the preferred post-edit and rejected legacy NMT output without requiring the optimizer to increase the absolute probability of the post-edit. This matters because the nominal winner is no longer assumed to be a desirable global target.

Second, the method adds a one-sided base-response anchor. For every training source, the upgraded model’s own pre-training response is available. The anchor penalizes training when that response becomes less likely than under the frozen reference model, but does not penalize increases in its likelihood. Existing behavior therefore receives explicit protection rather than being left to survive preference training by accident.

Third, token-level sequential KL regularization constrains divergence at each generation position. That is relevant when the useful correction is localized. A terminology choice, punctuation convention, or stylistic adjustment may warrant change in one part of the sequence without justifying broad movement elsewhere.

The three components have distinct jobs: preserve the relative correction signal, protect the upgraded model’s existing response, and limit how broadly probability mass can move.

The ablations make the interaction the main result

The component experiments are more informative than a simple comparison with DPO.

On English-to-Hindi, SFT, DPO, APO-Down, TDPO, and BAPO individually remain close to the base model in all-pass performance. Combining downward directionality with anchoring in BAPO-Down raises all-pass by 7.4 percentage points. Adding token-level KL in the full StalePO objective raises the gain to 14.9 points.

That pattern does not show that any one safeguard solves stale feedback. It supports the paper’s claim that the controls are complementary.

The category results also show what changed. Under automated MQM evaluation, English-to-Hindi Style improves by 20.9 points and Fluency by 16.5 points, while Accuracy declines by 5.4 points. Professional-translator evaluation on 2,000 English-to-Hindi segments points in the same direction: all-pass rises 13.8 points, Style 15.9 points, and Fluency 37.3 points, while Accuracy falls 3.5 points.

The human evaluation strengthens the English-to-Hindi result because it reproduces both the gain and its concentration. It does not remove the trade-off in accuracy.

For English-to-Turkish, the automated all-pass gain is smaller at 4.6 points, with positive changes in Style, Fluency, Accuracy, and Terminology. No professional-human validation is reported for that language pair.

Evaluation can become stale along with training data

The appendix exposes a second operational risk. If legacy post-edits are historically conditioned enough to be questionable training targets, they can also be questionable evaluation references.

A model trained to reproduce their vocabulary and syntax may improve BLEU, chrF, TER, or COMET against those references without delivering the largest improvement under an independent quality rubric. In this experiment, that is exactly the ranking disagreement observed between SFT and StalePO.

For localization teams, the Cognaptus inference is straightforward: after a material model upgrade, feedback provenance should become part of evaluation design. A reference created to repair generation \(N\) should not automatically authorize generation \(N+1\) merely because it was human-edited. Independent rubric-based evaluation or fresh human review becomes more valuable when the reference itself carries legacy-system structure.

The method needs residual signal to exist

StalePO has a specific operating region.

If the upgraded model is already better than the old post-edits on every relevant quality dimension, the paper provides no useful preference advantage for StalePO to extract. Reusing those examples would add stale evidence without residual information.

At the other extreme, if the human post-edits remain uniformly better than the new model, the stale-preference premise does not hold. A downward-biased objective is then poorly matched to the data because the preferred response really is a target worth moving toward.

The empirical certainty is also bounded. Most full-scale evaluation relies on Claude Sonnet 4.5 as an LLM judge; professional validation is limited to English-to-Hindi. Reported variability comes from five inference runs of a single training run rather than independently trained models, and company policy prevents disclosure of absolute MQM pass rates.

These constraints leave open how broadly the method transfers across models, languages, preference domains, and repeated training runs.

Preserve the upgrade, extract the remainder

The practical value of StalePO is a change in how historical human feedback is classified after a model upgrade.

Old corrections need not be either trusted as gold or discarded as obsolete. When they still expose specific deficiencies in the new system, they can be treated as partial evidence whose influence is constrained by the capabilities the new model already possesses.

That requires three decisions to stay separate: what direction the historical preference still supports, what current behavior must be preserved, and how locally the model should be allowed to change. The paper’s ablations suggest that solving only one of those decisions leaves much of the available improvement unrealized.

Just as importantly, the same historical artifact should not quietly serve as both the source of the optimization signal and the unquestioned definition of success. Once the model changes, the feedback—and the metric built around it—may need to be requalified.

Cognaptus: Automate the Present, Incubate the Future.