The Disagreement Worth Labeling
TL;DR for operators When two responses receive opposing preference judgments for helpfulness and safety, removing the disagreement may discard valuable training information. In a controlled study of multi-objective alignment, excluding conflicting comparisons reduced average reward in all nine tested model-dataset settings. The proposed Objective-conditioned Gradient Coverage DPO (OGC-DPO) method changes what an annotation team selects: not simply a response pair, but a response pair and the particular objective worth judging. It prioritizes judgments expected to influence the current policy while avoiding redundant supervision within each objective. ...