The Trace Has the Answer, Not the Alternative: Agentic-DPO for Offline Agent Training

TL;DR for operators

Historical agent traces usually tell you what a successful operator or agent did. They do not tell you which plausible alternative the current model is most likely to choose incorrectly. That missing contrast is the problem Agentic-DPO targets.¹

Instead of sending the student through full environment rollouts, Agentic-DPO pauses at states already present in expert trajectories, samples several one-step actions from the current student, and contrasts the expert action with a plausible different action the student actually favors. On StableToolBench with Qwen3.5-2B, plain SFT reaches 57.1% canonical accuracy while Agentic-DPO reaches 90.9%.

The result is not evidence that preference optimization alone solves agent post-training. The paper’s ablations show that the surrounding training design is load-bearing: without SFT warm-up, performance falls to 0.0% in the reported parser-validity ablation; removing the continuing SFT anchor cuts canonical accuracy sharply; removing the augmentation components substantially increases sensitivity to tool-schema perturbations.

For organizations with high-quality historical traces, the practical opportunity is to extract more decision-level supervision from logs before paying for repeated online rollouts. The boundary is equally concrete: Agentic-DPO never leaves the state distribution represented by those expert trajectories. Rare failures, changing environments, and workflows in which the student’s own mistakes create new downstream states remain candidates for online interaction.

Expert traces record the right action, not the student’s likely mistake

Supervised fine-tuning treats an expert trajectory primarily as a sequence of positive targets. At a given state, the model learns that the expert action should receive higher probability. What it does not explicitly learn is which competing action matters most.

That omission becomes relevant for agents because decisions are structured. A tool-using model may have several syntactically valid actions available, only one of which advances the task correctly. Training against an arbitrary wrong action would provide weak information if the current policy would never select it.

Agentic-DPO therefore constructs what the paper calls a state-conditioned action preference. At each expert decision state, it samples four one-step candidate actions from a snapshot of the current student. Invalid actions and actions equivalent to the expert choice are removed. Among the remaining candidates, the highest-probability different action becomes the hard negative.

The preference pair is consequently specific to both the state and the current policy:

expert action > plausible student mistake

The method then applies a DPO-style loss to increase that relative preference against a frozen SFT-warmed reference policy. Only action-token probabilities enter the comparison. Longer action strings receive a scaled preference coefficient so that length alone does not dominate the update.

This construction changes what existing traces can teach. The trajectory supplies the correct decision; the current student supplies the competing decision that currently deserves correction.

The student supplies the negative without generating a new trajectory

This is where Agentic-DPO can be confused with a cheaper version of online reinforcement learning.

It is not. The sampled alternatives are generated at expert states and are not executed in the environment. There is no student trajectory that drifts through a sequence of self-induced states during the gradient step.

That restriction explains both the cost advantage and the coverage limit.

In the paper’s tau-bench comparison with Qwen3.5-9B, Agentic-DPO reaches a 41.4% success rate versus 40.0% for GRPO, with overlapping reported uncertainty. The wall-time comparison puts Agentic-DPO at about 1.6 times the cost of an SFT step, compared with 13.6 times for GRPO.

The relevant conclusion is therefore narrower than “offline DPO replaces RL.” Under the tested configuration, the researchers recover comparable tau-bench performance while avoiding the full online rollout machinery used by GRPO during optimization.

The same method is strongest across the paper’s three principal benchmark families. With Qwen3.5-9B, it reports 94.1% canonical and 92.0% perturbed StableToolBench accuracy, 41.4% tau-bench retail success, and 64.4% Mind2Web average step success. On Mind2Web, that compares with 45.6% for SFT and 56.8% for ETO.

Across the three Qwen3.5 scales, the reported average improvement over plain SFT is 25.3, 22.2, and 18.0 percentage points for 2B, 4B, and 9B respectively. These are benchmark comparisons rather than causal estimates of what an arbitrary production workflow should expect.

Preference training works here because imitation remains in the loop

The ablation table is particularly informative because it tests whether the preference mechanism can stand on its own. It cannot in the reported setup.

Removing SFT warm-up produces 0.0% canonical and perturbed performance for both Qwen3.5-2B and Gemma3-4B in the ablation. The paper attributes this failure to action-format validity: preference optimization needs a policy that can already operate inside the required action schema.

The second stabilizer remains active during preference training:

Removing that SFT anchor lowers Qwen3.5-2B canonical accuracy from 90.9% to 72.5%. On Gemma3-4B it falls from 76.5% to 62.5%.

The paper’s local theoretical analysis gives a limited explanation for why this combination is compatible rather than contradictory. At initialization from the SFT reference, the DPO contrastive gradient points in the same logit-space direction as the SFT gradient under the paper’s simplified latent-action model. This is an initialization-level argument, not a proof of global convergence.

For an operator evaluating the recipe, the empirical evidence is more immediately consequential: the imitation component is not merely pretraining that can be discarded once preference optimization begins. It continues to constrain the policy while the negative examples push decision boundaries.

Schema robustness requires preserving the decision while changing its rendering

Agent actions contain another source of noise: textual form.

Tool names can change. Arguments can be reordered. Serialization formats can differ. Context can present the same operational state with different surface structure. A preference learner can mistakenly optimize these rendering differences instead of the underlying choice.

Policy-Preserving Augmentation, or PPA, addresses this by generating alternative renderings that retain the same latent expert decision. The paper uses both action-rendering rewrites and recovery-context rewrites.

The ablations indicate that these transformations mainly matter for robustness rather than raw canonical competence. Removing action-rendering PPA leaves Qwen3.5-2B canonical accuracy at 88.5%, but perturbed accuracy drops to 75.3%, producing a 13.2-point gap. Removing recovery-context PPA produces a 10.7-point gap. The full recipe’s gap is 5.4 points.

A separate held-out perturbation test serves as a robustness check rather than a second performance thesis. Agentic-DPO reaches 89.6% accuracy on two perturbation operators related to its training augmentations and 86.9% across seven operators held out from PPA. That supports some generalization beyond the exact transformations seen during training, although the evaluation remains a synthetic StableToolBench perturbation suite.

The zero-shot ToolACE-to-BFCL-v3 experiment points in the same direction. Agentic-DPO scores 84.17 on Non-Live and 57.18 on Live evaluation, compared with 81.87 and 51.54 after SFT. The Base model scores 82.66 and 56.11, making this result particularly useful: ordinary SFT does not automatically improve the out-of-distribution benchmark, whereas Agentic-DPO modestly exceeds the base model on both reported measures.

Existing traces become more valuable only when they cover the states that matter

The business case follows from the training geometry rather than from benchmark rank alone.

For an organization already collecting high-quality expert or operator trajectories, Agentic-DPO suggests a way to reuse those logs twice. The expert action supplies the positive target, while the current student supplies a policy-relevant competing action. This may let a team obtain decision-level corrective supervision without constructing a reward model or operating a full rollout environment for every optimization step.

The evidence also suggests that the method can be relatively data-efficient. In the StableToolBench scaling study reported by the paper, Agentic-DPO using 25% of the expert trajectories already matches the performance of PPA+SFT trained on the full dataset. Performance from negative sampling also saturates quickly around two to four candidates.

Cognaptus inference: for workflows whose critical states recur and are well represented in historical traces, the first post-training investment need not automatically be more demonstrations or online RL infrastructure. Reconstructing student-specific alternatives at those known states may extract substantially more supervision from the same logged operations.

The qualification is structural. Agentic-DPO cannot observe a state that exists only because the deployed student made an earlier mistake. If a wrong tool call changes a database, triggers an unusual recovery path, or places the agent into a context absent from expert demonstrations, the offline preference construction has no state from which to learn that recovery.

That makes state coverage the deployment test. Stable, repetitive workflows with strong trace coverage fit the method’s assumptions better than environments defined by rare failures, evolving interfaces, or consequential exploration.

Reproducibility is another current boundary. The source package records that the linked implementation presently covers the StableToolBench Qwen3.5-2B training path and canonical pairs, while several evaluation pipelines and trained checkpoints described in the paper are not yet fully released. That does not alter the reported paper results, but it affects how much of the full benchmark claim an external team can independently reproduce from the current artifact.

The decision is whether to optimize known states before exploring new ones

Agentic-DPO makes a specific argument about where additional supervision can come from. An expert trace already contains the correct decision. The missing information is the plausible alternative favored by the model being trained.

Sampling that alternative locally produces a stronger corrective signal than positive-only imitation in the paper’s experiments, and it does so without the rollout cost of online RL during gradient steps. But the same local construction prevents the method from discovering situations outside expert coverage.

For organizations deciding how to post-train an agent, that creates a practical sequence. When logged trajectories densely cover the operational states that matter, extract more policy contrast from those traces first. When failure states emerge from the agent’s own behavior or the environment keeps moving, offline optimization reaches its natural limit and new interaction data becomes necessary.

Cognaptus: Automate the Present, Incubate the Future.