Relevant Is Not Executable

TL;DR for operators

The routing mistake to worry about is not only sending a request to an irrelevant agent. It is sending the request to an agent that understands the topic but lacks the tools, authority, live data, or service scope required to complete the task.

Zhang, Cheng, and Li’s Debate-to-Skill paper¹ trains directly on that distinction. Its raw labels separate executable pairs from related-but-capability-insufficient or risky pairs and irrelevant pairs; after deployment-oriented collapse, only executable pairs count as positive. The method then supervises the reasoning used to reach that verdict through retrieved capability principles, opposing semantic and capability arguments, judge aggregation, and verifier-based reinforcement learning.

On 312 difficult related-but-not-executable examples, Debate-to-Skill raises F1 from 45.8 for direct label SFT to 64.9 while cutting over-acceptance from 33.7 to 16.9. Ablations weaken when debate, dynamic skills, or the judge is removed. The authors also report four-week online lifts of 15.79% in card impressions, 7.63% in card CTR, and 6.12% in redirect CTR, all with reported $p<0.05$.

For operators, the practical idea is narrower than “use multi-agent debate.” If routing errors come from confusing topical fit with executable capability, make that capability conflict visible during supervision. The debate runs offline, so the reported approach does not require multi-agent deliberation in the online serving path.

A relevant agent can still be the wrong destination

A user asks an agent system to do something. The router must decide which available agent should receive the request.

Semantic matching handles the easiest part of that problem. A travel agent may clearly match a question about flights. A finance agent may clearly match a question about a stock. But relevance does not establish that the selected agent can perform the requested action.

The missing distinction appears when the request is related to the agent but execution fails for another reason: the agent lacks a required tool, cannot access real-time information, lacks authority to perform an action, or operates outside the relevant service boundary.

The paper encodes this explicitly. Raw label 2 means executable, label 1 means related but capability-insufficient or risky, and label 0 means irrelevant. For deployment, only label 2 becomes positive.

That middle category is the paper’s difficult case. Calling it a grey zone is appropriate only after recognizing what makes it grey: semantic evidence points toward acceptance while capability evidence points away from it.

Debate supervises the capability decision, not the serving path

A richer label taxonomy alone would not necessarily teach a model how to distinguish those cases. Debate-to-Skill adds structure to the supervision process.

For each query-agent pair, the system retrieves the top three relevant capability principles from a skill bank using lexical cosine similarity. One side of the deliberation develops the semantic-support case. Another develops capability objections. A judge resolves the disagreement with guidance from the retrieved principles.

Recurring conflicts are then used to update the skill bank. The update is not unrestricted self-expansion: disagreement patterns enter a priority buffer, and only recurring high-priority patterns can add or refine a principle.

Training combines reasoning SFT with verifier-based GRPO on Qwen2.5-14B-Instruct. The verifier reward emphasizes verdict correctness while giving smaller bonuses for required-section validity and valid skill use.

This distinction matters operationally: the deliberation machinery produces supervision offline. The deployed router does not need to run a multi-round debate for each user request.

The largest gain appears where semantic similarity is misleading

The main offline comparison supports the paper’s central framing.

Moving from direct-label SFT to generic reasoning supervision already helps. Debate-to-Skill improves further: Main Test accuracy rises by 10.3 points over SFT Label, while grey-zone F1 rises by 19.1 points.

The pathology analysis is more informative than the aggregate score. Grey-zone over-acceptance falls from 33.7 to 16.9. Positive miss rate also falls, from 9.8 to 7.4.

So the improvement is not produced merely by rejecting more requests. The reported system becomes substantially less willing to accept capability-insufficient matches while also slightly reducing missed executable matches.

The harder traffic slices move in the same direction. Debate-to-Skill reaches 77.4 accuracy and 79.9 F1 on the professional-domain set, and 72.2 accuracy and 78.6 F1 on the long-tail set. These tests support persistence beyond the main benchmark, although they remain within the same proprietary industrial environment.

The ablations show why the process matters

The structural ablations help distinguish the proposed mechanism from a generic “more reasoning is better” interpretation.

Removing debate reduces grey-zone F1 from 64.9 to 56.7. Removing dynamic skills lowers it to 59.8. Removing the judge produces the largest drop, to 49.6.

The process audit sharpens that result. Full Debate-to-Skill records 88.3 for capability-conflict resolution. Without debate, that falls to 72.4; without dynamic skills, to 79.1; without the judge, to 58.6.

One result is especially revealing. The variant without dynamic skills achieves 97.5 structured validity—higher than the full system’s 88.1—yet its skill-grounding rate falls to 62.8 from 92.7 and its routing performance deteriorates.

Formatting the reasoning correctly is therefore not sufficient evidence that the capability logic is doing its job. The useful signal appears tied to grounding, conflict exposure, and resolution rather than surface compliance with a reasoning template.

Production results provide a pathway, not a complete audit

The paper also reports a four-week production A/B test with 10% treatment traffic. Relative to the deployed SFT Label baseline, the new routing policy produces:

• +15.79% card impression rate
• +7.63% card CTR
• +6.12% redirect CTR

All three are reported with $p<0.05$.

These results connect the offline routing gains to production behavior, but their interpretation has a firm reporting boundary. Online traffic volumes are not disclosed. Neither are the absolute metric levels or interval estimates.

That prevents an external reader from assessing baseline scale, practical variance, or the precision of the reported relative lifts. The production evidence is aligned with the offline results; it is not independently auditable from the information reported.

Cognaptus inference: supervise the failure boundary you actually care about

The business implication is not that every routing system should reproduce this exact debate architecture.

The narrower inference is more transferable. If a router’s costly errors arise because semantic relevance is being mistaken for executable capability, then capability failure should become an explicit supervision object.

That changes several operating decisions.

Training data can preserve related-but-non-executable cases instead of collapsing them into generic negatives too early. Repeated disagreement patterns can be reviewed as candidate operating principles. Cases where semantic and capability signals remain unstable can be sent for recheck or human review. Richer offline reasoning can also be used without automatically increasing serving latency.

The affected user is the person relying on the routing layer to surface an agent that can complete the requested work. The decision is whether to accept a query-agent match. The relevant condition is a system where execution depends on tools, authority, live access, or service scope that semantic similarity does not reliably encode.

Outside that condition, the value of this supervision design is less certain.

What remains unresolved

The evidence is moderate to strong inside the reported setting, but several parts remain deliberately bounded.

Skill retrieval is lexical rather than learned or dense. Skill updates are rule-driven. The system does not provide a calibrated abstention threshold. Its verifier checks verdict correctness, required structure, and skill use, but does not establish that every natural-language reasoning step is faithful.

More importantly, the evaluation comes from one proprietary Query2Agent environment. The reported results do not establish equivalent gains on public tool-use benchmarks or unrelated agent architectures.

These are not side issues for deployment planning. They determine whether the method should be adopted as a complete architecture or treated as evidence for the more general design principle underneath it.

The routing target is executability

Agent routing can look like a relevance problem because relevance is easy to observe and easy to label. The harder production failure occurs when relevance is present but execution capability is not.

Debate-to-Skill makes that failure class explicit, supervises the reasoning needed to separate it from valid matches, and reports improvements precisely where the distinction is most difficult. Its strongest contribution is therefore not the presence of debate itself. It is the decision to train on capability boundaries rather than asking a relevance label to stand in for them.

For agent systems whose usefulness depends on whether the selected component can actually perform the requested action, that is the operational question worth measuring.

Cognaptus: Automate the Present, Incubate the Future.