Buy Fewer Labels, Ask Better Questions: RLHF as an Allocation Problem

TL;DR for operators

Preference labels are usually budgeted as a quantity: buy more comparisons, improve the model further. Efficient Exploration at Scale suggests that this accounting misses a major variable—the value of the next comparison depends on which model produced it and whether the preference is actually uncertain.¹

In the paper’s Gemma 9B pipeline, information-directed exploration reaches with fewer than 20,000 preference choices a win-rate level that offline RLHF requires more than 200,000 choices to reach. That is a directly observed efficiency improvement greater than 10x within the experiment.

The result should not be read as “uncertainty sampling saves 90% of labels.” The reported system combines three changes: it continually updates the reward model and policy as feedback arrives, adds a small positive reinforcement offset to keep those online updates from tanking, and then spends feedback on response pairs where the reward model is epistemically uncertain. The approximately 1,000x figure discussed at larger scale is different again: it comes from extrapolating fitted scaling curves, not from an experiment using one million versus one billion labels.

For post-training teams, the immediate design question is therefore not simply how many judgments to purchase. It is whether the feedback pipeline can dynamically decide which outputs deserve the next judgment.

Static feedback can become stale while the model improves

Offline RLHF collects preference data from a fixed response distribution and trains against that accumulated dataset. The underlying operational problem is straightforward: once the policy changes, some of the comparisons bought earlier may describe regions of behavior the current model rarely visits.

The paper tests whether moving the collection process along with the improving model changes feedback efficiency. Its online procedure incrementally updates both the reward model and language model while new responses are generated. Candidate responses therefore increasingly come from the policy that the system is currently trying to improve.

Within the shared experimental pipeline, online RLHF substantially outperforms offline RLHF as preference choices accumulate. This is main comparative evidence, not merely an implementation detail: the distribution from which feedback is collected appears to matter alongside the volume of feedback.

Cognaptus inference: organizations running repeated post-training cycles should treat the annotation queue as a dynamic resource. If the policy changes materially between rounds, continuing to label a frozen pool may be cheaper administratively while being more expensive per unit of model improvement.

That inference has a boundary. The study uses Gemma 9B, an internal prompt corpus, and simulated preference choices. It does not establish that every production pipeline will receive the same return from moving to continual annotation.

The greater-than-10x result comes from a system, not one sampling trick

The paper’s strongest observed result appears in Section 5.1. Information-directed exploration reaches a performance level attained by offline RLHF only after more than 200K preference choices while consuming fewer than 20K choices itself.

The tempting explanation is that the system simply finds uncertain examples and labels those. The experimental record is more complicated.

First, the policy is updated sequentially, shifting future data collection toward outputs produced by the improved model. Second, the strong online method retains an explicit learned reward model. Reward-model-free online alternatives tested by the authors improved on offline RLHF but were not competitive with the reward-model-based procedure. Those comparisons support the role of the reward model, although they should not be treated as perfectly isolated ablations because the authors report substantial trial-and-error algorithm and hyperparameter development.

Third, online updating creates its own failure mode: performance can tank during training. The authors address this with what they call an affirmative nudge, a small positive offset $\epsilon$ added to the reward-model reinforcement signal. In the reported comparisons, this prevents the collapse seen with otherwise similar updates without relying on the lower learning rates or checkpoint fallback approaches the authors examined.

So the observed efficiency gain is better understood as an interaction among three operating conditions: keeping training data relevant to the current policy, keeping incremental optimization stable, and then deciding which judgments are worth purchasing.

Reward uncertainty determines which comparison earns a label

Once the response distribution is current and online learning is stable, the next problem is selection. Not every pair of answers contains equal information.

The paper represents uncertainty using an epistemic neural network attached to the reward model. For each prompt, the exploration procedure samples 16 candidate responses. It then considers response pairs and measures disagreement across epistemic reward-model particles about which response should win.

The queried pair is the one maximizing variance in predicted preference probability:

In plain language, the system buys a judgment where its reward-model ensemble disagrees most.

Qualitative response examples serve as exploratory support for this mechanism: maximum-uncertainty pairs tend to contain more substantively different alternatives than minimum-uncertainty pairs. They help explain why the selection rule could increase information per label, but they are not independent quantitative proof of the 10x effect.

There is also a precise scope condition. The uncertainty being modeled belongs to the reward model, not the language model, and the algorithm chooses among responses for prompts already supplied. It does not actively decide which prompts deserve annotation.

The 1,000x number is a forecast, not a measured saving

The observed result is already large enough to affect annotation economics. The paper then asks whether the advantage could grow with scale.

Section 5.2 fits a power-law-style relationship between accumulated preference choices $n$ and win rate:

Extrapolating those fitted curves implies that an efficient-exploration run using roughly one million labels could match performance attributed to about one billion offline-RLHF labels—approximately a 1,000x efficiency difference.

This is a scaling projection. The authors did not run the corresponding one-million-versus-one-billion-label experiment.

Capacity planning should use the first row as experimental evidence and the last row as a hypothesis about future scale.

Post-training budgets should include an allocation policy

The business opportunity is not merely reducing an annotation invoice. It is changing what an annotation system does between judgments.

For a team repeatedly improving a model, the paper points toward a feedback loop in which the current policy generates candidate behavior, the reward model identifies unresolved comparisons, expensive reviewers adjudicate those comparisons, and both models update before the next allocation decision. Annotation becomes part of the learning policy rather than a dataset acquired in advance.

That architecture is most relevant when preference judgments are genuinely costly—expert review, specialized domain evaluation, or other settings where wasted comparisons have material budget consequences. If labels are cheap, abundant, or collected asynchronously for reasons unrelated to current model behavior, the additional machinery may have less economic value.

The unresolved question is how much of the reported efficiency survives contact with production conditions. Feedback here is simulated by a Gemini 1.5 Pro-based preference model trained on real human feedback rather than supplied live by human raters. Evaluation centers on one model family and a proprietary prompt distribution. Multiturn conversations, agents with delayed consequences, and active prompt selection remain future extensions. The paper also states that its algorithm description is not sufficient for full reproduction.

Better RLHF may require buying different labels, not merely more labels

The paper’s most durable contribution is a change in the unit of optimization. Preference data is not treated as a homogeneous pile whose value rises only with size. Each new judgment is an allocation decision conditioned on where the current policy operates, where the reward model remains uncertain, and whether incremental optimization can remain stable.

The greater-than-10x result shows that this framing can materially alter the feedback-performance curve in a controlled Gemma 9B experiment. It is strong enough to justify testing adaptive annotation pipelines against static ones.

The approximately 1,000x projection is better treated as a research target than a budget assumption. The near-term decision is more concrete: before buying the next batch of preference labels, determine whether the system knows which comparisons are actually worth asking about.

Cognaptus: Automate the Present, Incubate the Future.