TL;DR for operators

When two responses receive opposing preference judgments for helpfulness and safety, removing the disagreement may discard valuable training information. In a controlled study of multi-objective alignment, excluding conflicting comparisons reduced average reward in all nine tested model-dataset settings.

The proposed Objective-conditioned Gradient Coverage DPO (OGC-DPO) method changes what an annotation team selects: not simply a response pair, but a response pair and the particular objective worth judging. It prioritizes judgments expected to influence the current policy while avoiding redundant supervision within each objective.

Under identical budgets of 1,800 objective labels, OGC-DPO leads average oracle reward in nine of nine settings, worst-objective reward in seven, and GPT-4o win rate in six. The strongest evidence concerns automated reward-model benchmarks, not actual human alignment.

For teams collecting expensive expert preferences, this supports testing selective objective-level annotation and retaining informative disagreements. Whether the approach saves money depends on estimator accuracy, independent evaluation, and whether annotation savings exceed additional computation.

When disagreement becomes a reason to delete useful data

Consider an assistant evaluating two responses to a potentially harmful request. One provides detailed information but creates safety concerns. The other refuses without offering a useful alternative. A helpfulness evaluator and a harmlessness evaluator may reasonably prefer different responses.

An alignment team has several choices. It can discard the comparison because the objectives disagree, request judgments for every objective, or determine which specific judgment would contribute most to improving the model.

The first option has a measurable cost in the study by Duc Anh Vu and colleagues, Objective-Conditioned Active Preference Learning for Multi-Objective Alignment.1 Their conflict-filtering ablation (Section 5.4, Figure 3) removes comparisons with opposing objective preferences while preserving the annotation budget. Average oracle reward falls in all nine model-dataset settings.

The qualitative evidence helps explain why. In an example involving a private-address request (Appendix E, Table 7), the reward difference between disclosure and refusal is +10.09 for helpfulness but −45.38 for harmlessness. The disagreement reflects a substantial objective trade-off.

Other disagreements are much less informative. Two substantially similar refusals receive reward differences of only −0.25 and +2.69 across the same objectives. Their opposing signs do not imply an equally consequential training decision.

These examples are illustrative, not a separate aggregate performance test. The central finding is narrower: automatically removing every disagreement can eliminate supervision that helps the model learn competing objectives.

That creates a selection problem. The team needs to identify which comparisons and which objectives justify another annotation.

The next judgment concerns an objective, not just a response pair

Traditional active preference learning often chooses a response comparison for labeling. Multi-objective alignment adds another dimension: the same comparison may be valuable for correctness but redundant for helpfulness.

OGC-DPO treats a candidate query as $(z,k)$, where $z$ represents a prompt and two responses, and $k$ identifies one objective. One judgment under one objective consumes one annotation unit. Labeling that pair under another objective consumes another unit.

The method uses three ingredients:

  1. Policy preference: How strongly does the current model implicitly favor one response over the other?
  2. Objective preference estimate: How likely is a criterion-specific estimator to prefer that response?
  3. Gradient coverage: Has the team already acquired judgments that would produce similar policy-update directions for that objective?

Each objective has its own LoRA preference estimator and accumulated gradient-coverage matrix. The projected policy-gradient direction is shared across objectives, but each objective has a different estimated preference discrepancy and coverage history.

The acquisition score can be written as:

$$ u_{k,t}(z) = |q_t(z)-p_{k,t}(z)| \,\|\phi_t(z)\|_{V_{k,t}^{-1}} $$

Here, $q_t$ is the current policy’s implicit preference probability, $p_{k,t}$ is the objective-specific estimator’s probability, and the final term measures how underrepresented the projected gradient direction is in that objective’s existing annotations (Section 4.2, Equation 8).

A query ranks highly when its estimated preference discrepancy is substantial and its direction is insufficiently covered.

This differs from ordinary uncertainty sampling. If both probabilities equal 0.5, the expected-gradient feature is zero even though the estimated preference label is maximally uncertain. OGC-DPO targets estimated mean policy change, not uncertainty by itself.

The distinction matters because a judgment can be uncertain without providing a strong expected update to the current model.

In each experimental round, the authors generate 3,000 candidate response pairs from 1,000 prompts and select 300 objective-specific queries. Six rounds produce 1,800 labels. The first round is randomly balanced across objectives to initialize the estimators. Subsequent selections use expected-gradient coverage, and the shared policy is trained with DPO and CAGrad-Clip.

Selecting a query updates coverage only for its chosen objective. The same response pair can therefore remain informative under another criterion.

Nine average-reward wins, but not nine wins on every metric

The main benchmark comparison (Table 1, Section 5.2) crosses three model families—Llama-3.2-3B-Instruct, Gemma-3-4B, and Qwen3.5-9B—with datasets containing two, four, or five alignment objectives.

OGC-DPO is compared with Random, ActiveDPO, and APO using identical objective-label budgets and a common policy-training procedure.

Evaluation metric Settings where OGC-DPO ranks first
Average oracle reward 9/9
Worst-objective oracle reward 7/9
GPT-4o win rate against initial policy 6/9

The advantage can be substantial, although its magnitude varies.

On PKU-SafeRLHF, Qwen3.5-9B achieves an average oracle reward of 10.72, compared with 9.37 for the strongest baseline, Random—a difference of 1.35 reward units.

For Llama-3.2-3B on the same dataset, the worst-objective reward reaches 7.66, versus 5.30 for the strongest baseline, APO. Its GPT-4o win rate reaches 76.4%, compared with 66.5% for the strongest baseline on that metric.

These are within-dataset comparisons. Reward scales differ across datasets, so the numerical improvements should not be treated as directly comparable effect sizes.

The worst-objective measure deserves particular attention. A method that improves aggregate helpfulness while severely degrading safety could appear successful under an average alone. The seven-of-nine result suggests that OGC-DPO often improves aggregate performance without sacrificing the weakest measured objective relative to its competitors—but not consistently in every setting.

The evidence supports more effective label allocation within this benchmark protocol. It does not establish that the resulting models are universally safer or more useful to humans.

The ablations test selection quality; the theorem establishes coverage

Two controlled variants identify different dependencies behind the result.

The conflict-filtering ablation tests whether disagreement should be excluded. Its consistent average-reward losses indicate that removing conflicting pairs can eliminate information that OGC-DPO would otherwise exploit. This does not prove every conflict is beneficial. Nor does the acquisition rule explicitly recognize opposing gradient directions: its coverage score is invariant to gradient sign.

The estimator warm-start ablation (Figure 6) tests whether active selection can operate effectively without first training its objective-specific preference estimators. On Llama-3.2-3B across the three datasets, warm-starting improves average reward in every examined round. Effects on worst-objective reward are less consistent, including negative differences on HelpSteer.

This makes estimator quality a material implementation dependency. Biased preference estimates can misdirect the annotation budget before the shared policy is trained.

The paper also proves that its fixed-feature, within-round log-determinant coverage objective is monotone submodular (Appendix B). Greedy selection achieves the conventional $1-1/e$ approximation bound for maximum coverage gain under the query-count constraint.

That guarantee concerns gradient coverage, not final reward, Pareto optimality, or human preference satisfaction. It justifies the acquisition procedure’s coverage efficiency, while the observed alignment improvements remain empirical.

The annotation savings still need a full cost and validation test

For organizations aligning assistants across safety, correctness, helpfulness, and other separately evaluated criteria, the business opportunity is selective supervision.

If expert judgments have approximately additive costs per objective, collecting a single targeted judgment can be less expensive than labeling every response pair against every criterion. OGC-DPO offers a principled way to select those judgments while retaining potentially informative conflicts.

Cognaptus would treat this as a candidate annotation workflow, subject to four measurable conditions.

First, objective-specific estimators must be sufficiently accurate to guide acquisition. Second, evaluation must independently assess each criterion, particularly the weakest-performing one. Third, disagreement filtering should distinguish consequential trade-offs from near-equivalent responses rather than rely on conflicting signs alone. Fourth, the total cost must include estimator training, forward passes, gradient projection, coverage maintenance, and policy optimization—not annotation counts alone.

The computational boundary is material. OGC-DPO requires extra objective estimators and, as Appendix C reports, $(K+1)n$ estimator forward passes for $n$ candidate comparisons. Different objective allocations may also change policy-training compute. Equal annotation budgets therefore do not guarantee equal end-to-end cost.

The evaluation boundary is equally consequential. Automated reward models generate the preference labels and supply the primary reward-based outcomes. GPT-4o comparisons offer an additional assessment, but direct human preference validation is not reported. The experiments cover three approximately 3B–9B-class models, English datasets, two to five objectives, and six acquisition rounds.

A production pilot would need independently judged examples, objective-level performance thresholds, estimator-calibration checks, and a complete annotation-plus-compute cost comparison. The economic case may weaken where labeling is cheap or one expert review can supply several objective labels at little additional cost.

Keep the conflict; choose the judgment

This research changes the unit of decision in active preference alignment. Instead of asking only which response comparison deserves attention, a team can ask which comparison-objective combination offers the most new supervision for its current policy.

The conflict-filtering results show why this matters: disagreement between objectives can contain information that an alignment process loses by enforcing consensus too early.

OGC-DPO provides a concrete selection mechanism and consistent average-reward improvements under controlled budgets. Whether those improvements translate into better human-aligned assistants at lower total cost remains an open empirical question.

For alignment teams, the defensible next step is selective annotation with independent evaluation—not automatic conflict removal.


Cognaptus: Automate the Present, Incubate the Future.


  1. Duc Anh Vu and Thong Nguyen and Hoang Pham and Ponhvoan Srey and Luu Anh Tuan (2026). Objective-Conditioned Active Preference Learning for Multi-Objective Alignment. arXiv:2611.05506. https://arxiv.org/abs/2611.05506 ↩︎