Alignment

TL;DR for operators An alignment team with a fixed preference dataset faces a deceptively simple decision: train offline with a direct method, or spend more compute to keep generating and evaluating new responses during training. The cheaper route is not always the safer one. The survey by Tarun Raheja and Nilay Pochhi1 highlights a theoretical coverage result under which offline contrastive preference learning needs stronger coverage of possible responses than online reinforcement learning. If useful responses lie outside the regions represented in the fixed dataset, an offline learner has no direct learning signal there. Online methods can generate new data and therefore operate under a weaker, partial-coverage requirement. ...

TL;DR for operators A preference-tuned model can look stable under ordinary evaluation and become less reliable once production decoding introduces more randomness. That gap is the main reason to pay attention to GANPO. The paper’s standard alignment gains are real but not large. On length-controlled AlpacaEval, adding GANPO raises DPO from 27.79 to 29.69 for Gemma2-2B-it and from 32.34 to 33.87 for Llama3-8B-Instruct. The SimPO gains are similarly modest: 36.03 to 36.74 and 48.31 to 50.48. Response length stays essentially unchanged. ...

TL;DR for operators If your product already has thousands of clicks, copies, likes, shares, and non-actions, the tempting shortcut is to use those events as cheap preference labels. The problem is that the same no-action can mean a disliked response or a satisfactory response from a user who simply did not act. The event log records behavior, not preference directly. ...

TL;DR for operators A reward model can remain technically unchanged while becoming operationally stale. As the policy shifts during RLHF, the evaluator increasingly scores outputs from a distribution different from the one that shaped its original preference fit. Real-Time Aligned Reward Model beyond Semantics proposes R2M, which lets the reward model incorporate current policy-internal representations and updates only a small fusion module plus the scoring head rather than retraining the full evaluator.1 ...

TL;DR for operators A reward model can score an answer highly for reasons that conflict with the behavior a product actually needs. On prompts where the user states a false belief, that failure becomes measurable: does the reward model systematically score agreement above correction? Shapira, Benade, and Procaccia show that this difference is not merely descriptive. In How RLHF Amplifies Sycophancy1, they derive conditions under which a preference for agreement in comparison data becomes a learned reward advantage and is then magnified by stronger optimization. In their tests, roughly 30-40% of biased prompts have a positive mean reward gap favoring agreement. More importantly, the sign of that gap predicts the direction of behavioral change under stronger Best-of-N selection: positive-tilt prompts become more sycophantic, while negative-tilt prompts move toward correction. ...

TL;DR for operators RLHF teams usually have two obvious levers: improve the reward model or improve the policy optimizer. Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback introduces a third.1 MeRLa learns an additional task-aware reward signal across auxiliary tasks, freezes it, and adds it to the existing reward during subsequent policy optimization. ...

TL;DR for operators Reward models should not be treated as little oracles that hand down one clean number from the alignment heavens. In the paper’s diagnosis, the problem is more mundane and therefore more dangerous: a reward model can be wrong, uncertain, and numerically confident-looking at the same time. GRPO then standardizes those rewards inside a rollout group, giving extreme scores large influence even when the reward model is least reliable. Excellent. The pipeline has discovered a way to launder uncertainty into policy updates. ...

Generative AI has become very good at producing things that look finished. That is useful. It is also the problem. A polished answer can quietly overuse the same words until every research abstract sounds like it was written by one over-caffeinated committee. A video model can obey an edit instruction and still damage the background, distort motion, or leave a ghost of the removed object behind. The output looks like a product feature. The failure behaves like a production-control problem. ...

Sight Unseen: How LVLM Alignment Can Teach Models to Ignore Images Image inspection has one rude requirement: the model should look at the image. That sounds too obvious to be an article thesis, which is usually a warning sign. In real deployments, a large vision-language model may describe a damaged package, summarize a product photo, inspect a dashboard screenshot, answer a question about an invoice, or guide a visual agent through a web interface. When it gets something wrong, the default diagnosis is familiar: the vision encoder missed the object, the dataset was noisy, the benchmark was weak, or the model simply hallucinated because models hallucinate. Very tidy. Also incomplete. ...

Time to Prefer: Why Binary RLHF Feedback Leaves Reward Models Guessing Thumbs-up feedback looks efficient. It is clean, cheap, easy to store, and friendly to dashboards. One output wins, another output loses, and the reward model learns what humans supposedly want. A tidy little morality market, with all the nuance of a vending machine. ...