Same Actions, Different Outcomes: The Cost of Tool-Schema Bias

TL;DR for operators

Changing how an agent calls tools can change its task success even when the underlying operations remain identical. In a controlled study, one model’s success fell from 78.5% to zero after tools were consolidated into a particular interface representation, then recovered to 75.2% under another merged representation. Across nine main open-weight models, fully merging tools reduced success by a median of 19 percentage points.

For teams deploying agents, tool grouping and argument conventions deserve the same evaluation discipline as model selection. Test candidate schemas on representative tasks, distinguish invalid calls from incorrect actions, and verify actual execution. Small task-specific samples can identify difficult interfaces. Prompt instructions and reinforcement learning may improve performance, but strong gains on familiar schemas do not establish robustness to future API changes or different workloads.

The Backend Stayed the Same. The Success Rate Did Not.

An engineering team consolidates dozens of individual functions into one dispatcher. The backend still performs the same operations, permissions remain unchanged, and users submit the same requests. From a conventional software perspective, the restructuring appears behavior-preserving.

For an LLM agent, it may be anything but.

In one controlled comparison, Gemma-4-12B completed 78.5% of tasks when operations were exposed through their original interface. When those operations were fully merged using a flat argument representation, its success rate fell to zero. Changing the merged representation to a union encoding restored success to 75.2%.

The executable actions had not changed. Neither had the evaluation tasks.

These results come from Yinhong Liu and colleagues’ Action-Space Shaping for LLM Agents: Measuring and Mitigating Tool-Schema Bias, which examines how alternative representations of the same operations affect agent behavior.¹

The authors call this phenomenon schema bias: variation in task success attributable to the tool interface through which an agent must express its actions.

The operational concern extends beyond one failed configuration. An enterprise could select a model, validate its workflows, and then lose substantial reliability after reorganizing its API catalog without changing a single business operation.

How the Experiment Separates Interface Design from Task Capability

The central methodological problem is equivalence. If two tool interfaces produce different results, how can researchers establish that the interface caused the difference rather than a change in what the system could execute?

Liu and colleagues construct an executable transformation framework that separates the native action space—the operations the environment actually supports—from the schemas through which an agent invokes those operations.

Nine transformation operators generate alternative interfaces. They cover three broad types of change:

• Tool partitioning: merging multiple tools or splitting tools into more specialized functions.
• Individual tool representation: changing argument nesting, names, descriptions, or argument order.
• Dependent-call protocols: requiring transactions, reference resolution, or schema discovery before completing an action.

Each transformed interface receives an execution adapter that decodes valid interactions back into native operations. Reference traces are checked to establish that they execute equivalent actions and reach equivalent environment states.

This design permits controlled comparisons without assuming that differently expressed calls are inherently different tasks.

The primary evaluation uses 2,748 queries across 12 synthetic domains and 168 native operations, with 7,818 required native calls. Nine open-weight models are tested against 32 schema variants, producing 288 model–schema evaluation cells. Two additional closed models are evaluated on representative variants.

Success is demanding: the executed native-action multiset must exactly match the task’s gold actions. This is stronger than checking whether a tool call is syntactically valid, although it does not establish that every possible interaction history is equivalent.

Under these conditions, changes in measured success can be attributed to the tested interface representations.

Merging Tools Changes More Than the Number of Choices

Fully merging tools reduces exact success by a median of 19 percentage points across the nine principal open-weight models. Splitting generally preserves performance closer to the original interface.

A plausible explanation is that consolidation moves complexity from tool selection into argument construction. A model may face fewer function names but must encode the intended operation inside a more complicated dispatcher.

Yet the experiments show that merging is not one uniform intervention.

Source: Appendix C.3, Table 6. Exact success on the same 2,748 synthetic queries per model–schema configuration.

Gemma-4-12B handles the union encoding far better than the flat representation. Qwen3.5-27B shows the opposite sensitivity to nested arguments: performance collapses despite remaining near its native baseline with flat or union encoding.

Neither model size nor the simple count of available functions explains these outcomes.

Further controlled comparisons reveal two additional concerns. Semantic grouping often outperforms random grouping, although exceptions remain. Mixing multiple argument conventions within one tool catalog also reduces success below the average homogeneous-schema performance in 16 of 18 matched comparisons, with bootstrap intervals excluding zero.

For an enterprise integration team, this suggests that independently reasonable API conventions can interfere when combined in one agent-facing catalog.

Schema choice can even change which model appears strongest. Among seven earlier-generation models, 23 of 31 transformed interfaces produce a different top-ranked model from the native-schema evaluation.

A benchmark therefore measures compatibility with a particular representation as well as the agent’s capacity to perform the underlying tasks.

A Correct Intended Action Can Still Become an Invalid Call

Aggregate success rates conceal several operationally different failures.

Under fully merged schemas, agents frequently generate calls that the interface rejects and then fail to recover. Transaction interfaces produce problems with required begin, write, and commit operations or their associated handles. Schema-discovery protocols can result in incorrect argument values even when the requested operation executes.

The paper distinguishes seven failure categories: livelock, transaction-handle errors, rejected-and-stuck calls, wrong dispatcher classes, under-execution, over-execution, and wrong arguments.

The categories matter because they imply different engineering responses.

A rejected call suggests that the agent’s output does not satisfy the interface contract. A wrong-class error indicates incorrect routing. A wrong-argument failure can be more consequential operationally because the environment may accept and execute the call without reporting an interface error.

The researchers also conduct a counterfactual test: what happens when an off-schema call is accepted if it can nevertheless be decoded into a valid native action?

In several merged configurations, this recovers substantial performance. Some apparent task failures therefore arise from incorrectly expressing an otherwise correct action.

This does not mean every rejected call contains a correct intention. Nor do the automatically assigned failure categories reveal the model’s internal reasoning. They are execution-level diagnoses, not validated psychological explanations.

For workflow owners, the actionable separation is between choosing the wrong operation, expressing the right operation incorrectly, and executing an incomplete or incorrect sequence. Each requires different monitoring and remediation.

Representative Tasks Are Better Schema Tests Than Descriptions Alone

Can an engineering team identify an agent-friendly schema without running its entire production workload?

The researchers compare methods for estimating schema difficulty.

Query-free probes measure properties such as model likelihood and schema-call compliance without requiring actual target tasks. These achieve mean within-model Spearman rank correlations of 0.52 and 0.57 with full-evaluation schema rankings.

Testing approximately 100 representative target queries per schema raises the mean correlation to 0.81. With 800 queries, it reaches 0.92.

The operational inference is specific: a modest task-grounded pilot can provide better evidence for choosing between interfaces than extensive inspection of their descriptions alone.

However, those correlations come from the evaluated synthetic task distribution. The paper also finds that difficulty rankings do not reliably transfer from synthetic queries to the external tau-squared-bench environments.

The relevant sample must therefore resemble the tasks that the deployed agent will actually execute.

Training Repairs Familiar Interfaces, but the Repair Has Boundaries

The researchers investigate whether schema sensitivity can be reduced through instructions, constrained decoding, supervised fine-tuning (SFT), or on-policy reinforcement learning using Group Relative Policy Optimization (GRPO).

Variant-specific instructions substantially improve some difficult configurations. On Qwen3-4B, instructions raise fully merged success by 28 percentage points and schema-discovery success by 45 points relative to their respective baselines.

Constrained first-call decoding also helps selected configurations, but its effects are inconsistent.

Training produces larger changes under certain conditions.

For Qwen3-4B, mixed-schema GRPO improves fully merged success from approximately 36% to 79%, transaction success from 28% to 62%, and schema-discovery success from 7% to 75%.

Unlike mixed-schema SFT, which lowers native-schema performance by eight percentage points in the main comparison, mixed-schema GRPO increases native performance by four points while improving the targeted variants.

The intervention uses environment execution feedback, rewarding successful native-action execution while accounting for partial action recall, unnecessary actions, and protocol violations.

But the results depend on the training setup. In one schema-discovery ablation, mixed training without a KL penalty reaches 75% success, compared with 25% when the KL coefficient is 0.01.

More importantly, learning the conventions of observed interfaces is not equivalent to becoming invariant to arbitrary interfaces.

In the SFT schema-transfer experiments, represented transformations improve by 14.6 points, new combinations of familiar transformations by 8.8 points, and stronger versions of familiar changes by 11.2 points. Previously unseen operator types instead show an average decline of 2.4 points.

The strongest gains also weaken under external validation. Two SFT-then-RL training seeds improve synthetic performance by 9.2 and 10.7 points, but change task-weighted tau-squared-bench performance by only +1.1 and −0.7 points, respectively.

These are transfer tests, not evidence that training failed universally. They establish a narrower result: success on trained schema structures is a poor guarantee of reliable adaptation to new interaction conventions or workloads.

API Refactoring Should Trigger Agent Revalidation

The main synthetic environment contains typed operations, fixed reference actions, and single-utterance tasks. External tests on airline, retail, and multi-turn function-calling benchmarks support the existence of schema sensitivity, but cover only selected interfaces.

Absolute tau-squared-bench scores also cannot be compared directly with its official leaderboard because the tested model serves as user simulator and natural-language judge.

These boundaries limit claims about universal optimal schema designs and production improvements from training. They do not eliminate the engineering problem demonstrated under controlled conditions.

For enterprise platform teams, Cognaptus draws three operational implications.

First, tool-schema migration should be treated as a potentially behavior-changing release. Even when backend operations remain equivalent, changes to grouping, argument conventions, or dependent-call protocols warrant renewed agent testing.

Second, model selection should use the interfaces intended for deployment. A ranking obtained under one canonical tool catalog may not survive a different but equivalent representation.

Third, reliability monitoring should observe decoded operations and resulting states, not just successful tool-call submission. Parser rejections, repeated attempts, incomplete transactions, and silent parameter errors require separate detection.

These measures can be applied before investing in additional training. Where a difficult interface cannot reasonably be redesigned, schema-specific instructions or post-training offer possible remedies, provided their gains are verified on the target workload.

The study establishes a clear measurement result: agent capability is partly conditional on the interface through which that capability must be exercised.

For organizations building tool-using agents, API design is therefore part of the reliability specification. The relevant question is not whether two schemas expose the same functions, but whether the chosen model can execute the required work through each schema consistently.

Cognaptus: Automate the Present, Incubate the Future.