Better Commands, Missing Capabilities

TL;DR for operators

An agent’s ability to complete a desktop task depends on more than the model operating it. The commands exposed by the application, the effort required to discover them, and the ability to inspect saved results can materially change performance.

ANIS, an Agent-Native Interface Synthesizer, generates agent-oriented commands from existing software capabilities and validates their effects on persistent artifacts.¹ On 368 desktop tasks, DeepSeek V4 Flash improves from 68.5% success with native SDK access to 74.5% with ANIS, while average online token consumption falls from 84.4K to 21.2K per task.

The improvement comes with a qualification: ANIS performs worse than native SDK access in Blender and FreeCAD, where generated commands can miss operations involving complex runtime dependencies. Interface generation also requires 17.1–36.4 hours per application.

For automation teams, the actionable implication is to evaluate generated interfaces by capability coverage, verified artifact outcomes, online efficiency, and setup costs together. The reported success rates use three attempts per task and do not establish production reliability or total-cost savings.

An Agent Can Finish Executing Without Finishing the Work

Consider an agent instructed to modify a spreadsheet, save it, and return the completed file. It may execute every command without an error and report that the task is complete. Yet the saved spreadsheet can still contain the wrong formulas, formatting, or cell values.

The difficulty begins before the final verification. The agent must identify appropriate application operations, understand their parameters, combine them in the right order, and determine whether each operation changed the intended content. A capable language model can still struggle when those operations are poorly organized or the application offers no direct way to inspect the resulting state.

This creates a separate engineering problem for teams deploying agents into existing desktop software. Improving model reasoning or supplying more documentation does not necessarily make the application’s capabilities easier to execute correctly.

The research investigates whether changing the interface itself can improve task completion without rebuilding the application.

ANIS Builds Commands from What the Application Can Actually Do

ANIS approaches desktop automation by examining the installed application’s runtime capabilities rather than constructing an interface solely from documentation or presumed workflows.

It discovers operations through registered application functions, runtime inspection of objects and services, and application menu actions. From this evidence, it generates executable commands, tests their behavior, and retains commands whose effects can be validated against saved and reopened artifacts.

The result has two levels.

Atomic commands expose individual meaningful operations, organized around the artifact objects they manipulate. An agent can discover available functionality without first reconstructing the application’s broader SDK structure.

Goal-level commands combine validated atomic operations into reusable actions. ANIS identifies recurring command sequences appearing in at least three requests, determines whether they represent a common artifact-level subgoal, parameterizes them, and validates execution and saved results on clean artifact copies.

This architecture addresses three interface properties:

• Discoverability: The agent can locate relevant operations and understand their arguments without extensive exploration.
• Granularity: Commands can express meaningful operations or recurring objectives without requiring excessive low-level composition.
• Observability: Where appropriate state-reading functions exist, the agent can inspect persistent artifact content rather than relying on successful write-command execution.

The last condition is particularly consequential. ANIS can expose existing readback capabilities, but it cannot guarantee observability where the underlying application provides no suitable accessor.

Software Releases Changed Outcomes Without Changing the Agent

Before evaluating ANIS, the authors examined whether software-interface evolution affects agent performance.

They assembled 255 artifact-verified tasks across six applications—QGIS, Writer, Blender, Impress, Inkscape, and FreeCAD—and tested four historical releases of each application using DeepSeek V4 Flash and Claude Haiku 4.5.

Task instructions, starting artifacts, the OpenCode agent scaffold, execution conditions, and deterministic verifiers remained fixed within matched comparisons.

The newest release outperformed the oldest in all 12 model–application combinations.

In QGIS, DeepSeek’s success increased from 41% to 69%, while Haiku’s increased from 33% to 65%. The number of QGIS processing operations expanded from 272 to 440 across the evaluated releases. For Haiku, mean operation-chain length declined from 30.4 to 21.2 operations among the comparable successful trajectories.

These results establish that application releases can materially affect an agent’s ability to complete the same benchmark task. They do not establish that discoverability or operation granularity individually caused the improvement. Each release incorporates multiple software changes, and trajectory comparisons are restricted to tasks successfully completed across all four versions.

That distinction matters because interface synthesis is motivated by the observed release differences, while the proposed design properties remain partly explanatory interpretations.

Synthesized Interfaces Improve Efficiency, but Coverage Determines the Ceiling

The main ANIS evaluation expands the benchmark to 368 tasks across eight applications, adding LibreOffice Calc and GIMP.

The authors compare native SDK access, CLI-Anything, ANIS atomic commands, and ANIS with reusable goal-level commands. The following results use pass@3: a task succeeds if at least one of three independent attempts produces an artifact satisfying its deterministic verifier.

Source: Section 5.3.2, Table 2; Appendix 9, Table 4. All success rates are artifact-verified pass@3 results.

With DeepSeek, atomic commands improve success by 3.8 percentage points over native SDK access while reducing online tokens by 73.2%. Adding goal-level commands raises the total success improvement to 6.0 points and the token reduction to 74.9%. Haiku also gains 6.0 success points with the full interface.

This suggests that reorganizing existing application capabilities can reduce the reasoning and exploration required to complete tasks.

However, command syntax alone does not account for the results. CLI-Anything performs substantially worse under the reported setup. ANIS exposes more than 20 times as many commands across applications covered by both systems, and the authors attribute 98.7% of CLI-Anything’s failed tasks to unavailable commands.

The relevant distinction is functional coverage, rather than an inherent disadvantage of command-line interfaces.

Reading the Saved Artifact Changes the Outcome

The paper also examines agents that announce completion despite producing artifacts that fail verification.

An audit identifies 581 such trajectories. Of these, 569 contain a detected checking action, but 153 lack an independent readback of the persistent artifact state.

The audit demonstrates a gap between checking behavior and verified completion. It does not establish that the absence of readback caused every observed failure.

A separate ablation provides more targeted evidence. The authors prohibit observation commands in the ANIS atomic interface and audit execution traces to confirm compliance.

DeepSeek’s overall success falls from 72.3% to 60.3%, a decline of 12.0 percentage points.

The effect varies substantially by application. Calc loses 30.3 points and Writer loses 19.7 points. GIMP, Impress, and Inkscape show no aggregate change.

This is stronger evidence that observation commands contribute to task success under the tested conditions. It is not evidence that every application benefits equally or that readback alone guarantees correctness.

For teams automating document and spreadsheet editing, the implication is specific: make persistent-state inspection available alongside modification operations, and verify the saved result against explicit requirements.

The Same Interface That Saves Tokens Can Remove Necessary Operations

The aggregate improvements conceal two counterexamples.

With DeepSeek and goal-level commands, Blender achieves 51.4% success through ANIS versus 54.3% through its native SDK. FreeCAD presents a larger gap: 77.4% versus 96.8%.

The authors associate these failures with operations involving implicit object relationships, selection states, and complicated runtime preconditions. Such capabilities may exist in the application but be omitted when the synthesis system cannot validate them reliably.

A generated interface therefore can simplify common workflows while excluding operations required by other tasks. For applications with complex object dependencies, command coverage may be more consequential than reduced exploration overhead.

Reusable goal-level commands introduce another qualification. The full-benchmark evaluation uses previously solved task trajectories to construct them. A separate held-out evaluation addresses this concern by synthesizing seven goal-level commands from 258 training tasks and testing them on 110 disjoint tasks.

Successful completions increase from 64 to 67, or 58.2% to 60.9%, while tokens per task decline from 22.8K to 21.7K.

This smaller, independent improvement supports some generalization of reusable commands, but does not establish equivalent benefits for workloads without recurring subgoals.

Interface Adoption Requires a Coverage Test and a Cost Model

For AI-platform architects choosing how agents access desktop applications, the results argue against automatically replacing native SDKs with generated interfaces.

A more defensible adoption process is to test representative tasks through both interfaces, inspect unsupported operations and their preconditions, and apply artifact-level verification to completed outputs. Workloads that require capabilities omitted by the generated interface may need continued native SDK access.

The economic comparison is also incomplete. ANIS requires 17.1–36.4 hours of offline generation per tested application. The approximately 73–75% online token reductions do not include this generation effort, subsequent interface regeneration, or maintenance following application updates.

For an organization running repeated, stable document-editing workflows, those costs may be amortized over substantial task volume. For an organization with infrequent workflows or frequently changing software, the same setup burden may be harder to justify. These are deployment hypotheses, not cost conclusions demonstrated by the paper.

The benchmark itself also has boundaries. Its deterministic verifiers capture specified artifact outcomes, not every aspect of user satisfaction or file usability. Pass@3 is not first-attempt reliability, and no long-term deployment experiment establishes performance under evolving enterprise workloads.

The appropriate acceptance criterion is therefore broader than a headline success percentage: sufficient command coverage, reproducible saved outcomes, representative unseen-task performance, and measured end-to-end operating cost.

Interface Design Is Part of Agent Engineering

ANIS demonstrates that agents can become more effective through changes to the interface connecting them to software, even without improving the underlying language model.

The strongest result is conditional. Synthesized commands improve benchmark success and online efficiency when they make relevant capabilities easier to discover, compose, and inspect. The same synthesis process can reduce performance when complex application capabilities disappear from the generated interface.

For teams building desktop-workflow automation, the interface deserves its own engineering and verification process. A command is valuable not merely because the agent can invoke it, but because it exposes the operations needed to produce—and verify—the requested artifact.

Reference

Cognaptus: Automate the Present, Incubate the Future.