Train the Advice, Not Just the Agent

TL;DR for operators

A multi-step AI agent may retrieve useful information, lose track of a task constraint, or continue searching after it already has enough evidence. A separate advisor can recommend the next action, but asking a larger model for feedback does not guarantee that the advice will improve task completion.

Caddie¹ trains an advisor using what happens after the acting agent follows its suggestions. On MuSiQue with supplied passages, a Qwen3-4B agent improves from 21.56% success without assistance to 46.88% with Caddie. A prompted Kimi K3 critic reaches only 29.13% in the same comparison. The acting model remains unchanged; reinforcement learning updates the advisor.

The results also show potential reuse across different acting models and customer-service tasks. But the gains depend on the model, workflow, retrieval environment, and intervention policy. Training requires repeated agent continuations from reproducible intermediate states, and some transferred critics reduce performance.

For operators, the relevant investment is a separately trained feedback capability that can be tested against prompted critique and direct agent improvement. Its value depends on incremental completed tasks, consultation costs, and the reliability of the environment used for training.

A stronger second opinion does not necessarily finish more tasks

Consider an automated research agent that has gathered several sources but remains uncertain about its next move. It might need another search, a correction to its reasoning, or a decision to stop gathering evidence and answer.

An advisor could provide that direction. The difficulty is training it without requiring someone to label every intermediate decision as correct or incorrect.

Caddie addresses this problem by rewarding advice according to the agent’s eventual task outcome. The paper’s main benchmark comparisons show why this training distinction matters.

Source: Section 3.2, Table 1. All results use Qwen3-4B as the acting model, with a separately trained Caddie critic for each setting. Means are based on eight seeded rollouts per task, covering 200 MuSiQue questions or 134 unseen ALFWorld games. Standard errors are omitted from this compact table.

The largest differences appear in MuSiQue with attached passages and in ALFWorld, where Caddie improves unassisted success by approximately 25–26 percentage points. Wikipedia retrieval produces a smaller improvement of 10.75 points, and Caddie’s advantage over prompted Kimi K3 narrows to 1.94 points.

These results support a specific conclusion: training feedback against downstream task outcomes can produce better assistance than prompting a larger critic, under the tested conditions. They do not establish that outcome-trained critics will outperform prompted feedback in every task or at equivalent computational cost.

How the critic learns without changing the acting agent

Caddie uses a generative critic: a separate language model that reads the agent’s instructions, previous actions, and environmental observations, then produces a diagnosis and advice for subsequent steps.

The training procedure compares different recommendations from the same intermediate state.

1. Sample an intermediate point from an existing agent trajectory.
2. Generate 16 candidate critiques from a Qwen3-4B critic.
3. Append each critique separately to the shared trajectory history and resume the frozen acting agent.
4. Evaluate whether each resulting continuation successfully completes the task.
5. Update the critic to favor recommendations associated with better outcomes relative to other candidates from that state.

In the main configuration, each critique receives one continuation. Its reward is therefore the observed terminal outcome, rather than a judgment about whether an individual recommendation sounds correct.

The optimization uses DAPO, a group-relative reinforcement-learning objective. It updates only tokens generated by the critic. The acting agent’s parameters, actions, and continuation tokens receive no gradient updates.

This design removes the need for reference critiques or step-level correctness annotations. It also changes what the advisor can learn. On a retrieval task, for example, requesting more evidence may initially appear unnecessary but ultimately improve the answer. Terminal-outcome feedback can reward that intervention without requiring a separate label for the search decision.

At inference time, the acting agent can request help through a call_critic tool. The main evaluations permit at most three consultations per trajectory. The agent can also proceed without consulting the critic.

The mechanism thus separates two capabilities: the acting model executes the workflow, while the trained critic supplies context-dependent guidance.

Advice can transfer, but the receiving agent matters

A critic trained exclusively through one model’s continuations might learn recommendations that only that model can use effectively.

Caddie’s cross-model experiment tests this concern directly. The researchers train critics using Qwen3-4B, then apply them without additional training to Qwen3-30B-A3B, Qwen3.8-27B, and Kimi K3.

Across six MuSiQue model-and-retrieval comparisons, the trained critics improve mean success over unassisted execution. Improvements range from 9.31 to 13.19 percentage points with attached passages, but only 0.81 to 6.19 points with Wikipedia retrieval.

For Kimi K3 using attached passages, success rises from 38.56% to 51.00%. This is evidence that a smaller, separately trained advisor can provide useful guidance to an unseen acting model.

The next transfer test moves beyond question answering. MuSiQue-trained critics are applied without retraining to DeepDive search and tau3 Retail and Airline customer-service tasks.

With Qwen3-4B as the acting model, the Wikipedia-trained critic produces the following results:

Source: Section 3.4, Table 3. These are out-of-domain comparisons using 64 DeepDive, 40 Retail, and 20 Airline tasks, respectively, with eight rollouts per task.

The Wikipedia-trained critic improves the unassisted mean in all six tested combinations of the two acting models and three unseen task families.

Transfer nevertheless has meaningful exceptions. On DeepDive with Qwen3.8-27B, the attached-trained critic reduces success from 48.83% to 44.53%. In tau3 Retail with that larger model, a prompted Kimi K3 critic reaches 59.69%, exceeding the Wikipedia-trained Caddie result of 58.12%.

These comparisons make reusable advice a plausible engineering capability, rather than a guaranteed property of the trained critic.

An advisor can complement an already optimized agent

A complementary experiment asks whether separate advice still helps after the acting model has undergone task-specific reinforcement learning.

The researchers first optimize the Qwen3-4B generator using a GRPO-based procedure. They then freeze the optimized generator and train a new critic using its continuations.

On MuSiQue with attached passages, the optimized generator achieves 49.69% success without a critic. Adding the newly trained critic raises success to 63.56%, a reported gain of 13.88 percentage points (Appendix E, Table 5).

This supports complementarity between improving an agent’s execution policy and improving the feedback available during execution.

It is not universal: in the Wikipedia setting, the critic trained with the original generator achieves a slightly higher mean than the critic trained with the optimized generator, 24.44% versus 23.38%.

The distinction matters for teams already investing in agent fine-tuning. Additional generator optimization is not the only available improvement path, but a newly trained critic is not automatically preferable to an existing one.

Consultation timing and training infrastructure constrain deployment

The method’s most significant deployment constraints appear outside the headline benchmark comparisons.

Intervention timing is consequential. A sensitivity experiment in Appendix F holds the critic configuration fixed while changing when advice is inserted. For Qwen3-4B on MuSiQue-attached with one continuation per candidate, success rises from 17.75% when intervention occurs at step 3 to 30.88% at step 4 and 33.69% at step 5.

These are diagnostic fixed-step comparisons, not the main three-call self-requested evaluation. They demonstrate timing sensitivity rather than proving which policy is generally best. Other selected experiments find fixed-step intervention outperforming self-call.

Advice usefulness depends on agent behavior. The paper’s descriptive analyses associate additional search guidance with better outcomes for the smaller generator, while the larger generator more often benefits from advice to answer using evidence already collected. Because agents choose when to seek feedback, these associations are not causal estimates of individual advice categories.

Training requires reproducible continuation states. The default training configuration evaluates 16 candidate critiques, each with one frozen-agent continuation, from a common intermediate state. Increasing the continuation count can reduce reward-estimation noise, but also multiplies execution costs. The paper’s five-continuation experiments show reduced later training fluctuations without establishing superior convergence efficiency.

For an enterprise agent using external APIs, files, or transactions, reliably restoring an intermediate state can be difficult. Successful training may require sandboxed tool execution, replayable environments, and transaction-safe branching.

A further limitation concerns what the critic actually learns. On self-contained mathematical tasks, outcome optimization can favor supplying a complete solution for the acting model to copy. Higher terminal success then need not imply that the critic has learned transferable guidance for multi-step interaction.

What operators should measure before adopting a critic

Cognaptus infers three priorities for organizations considering this architecture.

For research agents, measure the incremental completion rate of advice that helps the agent gather missing evidence or stop unproductive searches. Compare trained and prompted feedback within the same retrieval configuration and acting-model checkpoint.

For customer-service agents, evaluate whether advice improves final transaction success while preserving user requirements and system state. The tau3 results justify targeted experimentation, but the relatively small task sets and benchmark environments do not establish production reliability.

For agent platforms supporting multiple models, treat critic reuse as a testable configuration decision. Track outcomes by acting model, task family, and environment, alongside consultation frequency, inference latency, tool costs, and training-continuation expense.

The benchmark evidence does not establish end-to-end economic efficiency or superiority over scalar verification under matched inference procedures and compute budgets. Those remain open architecture choices.

The value of an advisor is what happens afterward

Caddie demonstrates that a separately trained natural-language critic can improve an unchanged agent by learning from the consequences of its recommendations. Its strongest contribution is not simply another review step, but a way to train that review step without intermediate correctness labels.

The gains across MuSiQue, ALFWorld, unseen models, and customer-service benchmarks make the approach worth investigating for multi-step automation. The negative transfer result, timing sensitivity, solution-copying behavior, and continuation costs define where the evidence stops.

For operators, an advisor earns its place in the workflow when its guidance measurably increases successful task completion under the actual model, tools, constraints, and costs of deployment.

Cognaptus: Automate the Present, Incubate the Future.