Preference Optimization

TL;DR for operators Historical agent traces usually tell you what a successful operator or agent did. They do not tell you which plausible alternative the current model is most likely to choose incorrectly. That missing contrast is the problem Agentic-DPO targets.1 Instead of sending the student through full environment rollouts, Agentic-DPO pauses at states already present in expert trajectories, samples several one-step actions from the current student, and contrasts the expert action with a plausible different action the student actually favors. On StableToolBench with Qwen3.5-2B, plain SFT reaches 57.1% canonical accuracy while Agentic-DPO reaches 90.9%. ...

Fine-tuning looks deceptively democratic. Every response token gets its little vote in the gradient. The commas, the boilerplate, the obvious connective tissue, the wrong kind of certainty, the genuinely task-bearing step in the middle of the answer: all are invited to update the model. A charmingly egalitarian arrangement. Also a rather efficient way to teach a model to forget things it used to know. ...

Chart review is the boring part of medicine, which is exactly why AI systems should learn from it. A clinical discharge summary does not fail only when it sounds clumsy. It fails when it tells a patient something that did not happen, invents a medication change, adds a procedure, misstates a timing detail, or turns a vague note into a confident medical fact. The prose may still be smooth. The bedside manner may even be excellent. Unfortunately, a hallucination delivered in fluent patient-friendly language is not safer because it has better manners. ...

Training data has a bad habit: the easiest examples talk the loudest. Anyone who has trained a model on preference pairs knows the scene. One answer is clearly grounded in the image; the other confidently invents an object, a color, or an action that is not there. The model learns the contrast quickly. Everyone applauds. The loss goes down. The dashboard looks obedient. ...

A list of individually excellent items can still be a terrible list. Ask anyone who has attended a conference with five brilliant speakers, no agenda, and three consecutive sessions on the same topic. Recommendation systems have the same problem. A conventional recommender can assign highly accurate scores to individual videos, products, or articles, then still assemble a repetitive, badly ordered, or strangely balanced feed. Each item wins its private competition. The user receives the collective consequences. ...

Jailbreaks are not polite enough to wait their turn. That is the awkward weakness in many safety-training pipelines. A model is attacked, patched, tested, and released. Then another attack appears, usually crafted with more creativity than the previous defense assumed. The safety team patches again. The benchmark improves. The real attack surface moves. Everyone calls this iteration, because “organized whack-a-mole with GPUs” sounds less respectable. ...

TL;DR for operators Most LLM safety systems still assume that, when a model sees a harmful request, the correct behaviour is refusal. That works until the attacker stops arguing with the prompt and starts interfering with generation itself. The paper behind this article, Strategic Deflection: Defending LLMs from Logit Manipulation, proposes SDeflection: a fine-tuning method that teaches a model to answer in a safe, topic-adjacent way rather than relying only on explicit refusal language.1 The model does not provide harmful instructions. It redirects the subject toward harmless information that is close enough to the original topic to survive attacks that try to force compliance-style openings. ...