Passing the Tests, Missing the Handoffs

TL;DR for operators

A multi-agent coding workflow can pass most repository tests while still failing at the interfaces where one agent’s code depends on another’s. AsynCodeBench¹ makes those handoffs explicit: it pre-registers directed software dependencies, checks them after integration, measures final dependency satisfaction with Asynchronous Dependency Pass Rate (ADPR), and records first resolution with Dependency Resolution Step (DRS).

The measurement changes practical conclusions. Across the benchmark’s protocols, 12.6% to 16.8% of model-task outcomes achieved at least 80% Test Pass Rate while resolving fewer than half of their declared dependencies. Larger models and additional coordination did not reliably close that gap. Five of ten models also changed their within-model quality-cost Pareto frontier when quality was measured by ADPR instead of conventional Test Pass Rate.

For organizations running coding-agent teams, repository tests should remain part of acceptance, but they are not sufficient evidence that independently generated components compose correctly. Dependency checks can inform release gates, agent decomposition, manager permissions, protocol selection, and token spending. The boundary is equally clear: these results come from 19 repository-grounded tasks and controlled benchmark protocols, not from a randomized study of industrial engineering teams.

A repository can look healthy while its handoffs are broken

Consider a coding task split among several specialized agents. One modifies an upstream component, another implements code that depends on it, and the integrated repository passes most of its tests.

That result looks reassuring until the downstream component is exercised specifically against the behavior the upstream agent was supposed to provide.

This distinction appears repeatedly in the benchmark. Across all five execution protocols, between 12.6% and 16.8% of outcomes combined at least 80% Test Pass Rate with less than 50% dependency satisfaction. In one DeepSeek-V4 Async-Manager run on TVM-20073, Test Pass Rate reached 85.7% while ADPR remained 0%.

The two measurements are related, but not interchangeable. Their Pearson correlations ranged from 0.599 to 0.716 across protocols, while Test Pass Rate exceeded ADPR in 62.6% to 77.9% of model-task outcomes.

The failure being exposed is therefore narrower than “the tests are bad.” Conventional repository tests measure task correctness over their available test points. They do not necessarily identify whether a particular producer-consumer relationship between independently implemented components has been satisfied.

AsynCodeBench measures the interfaces agents must actually share

AsynCodeBench contains 19 tasks from 16 public repositories, divided into 64 implementation units with 52 directed dependencies. The dependency graphs are defined before model evaluation and validated using repository code, interfaces, task decomposition, public tests, and independent human review.

Each directed edge represents a concrete requirement: one component produces behavior that another component consumes. Executable Dependency Checkers then test whether that requirement survives integration.

ADPR summarizes the final state:

In plain language, it is the proportion of declared dependencies that work in the final integrated workspace.

DRS asks a different question:

It records the first logical integration checkpoint at which dependency $d$ becomes satisfied. This is not wall-clock latency. The checkpoints represent ordered execution progress, allowing the benchmark to distinguish dependencies resolved early, resolved late, or never resolved.

That distinction matters because a final-state score cannot show how coordination developed during execution.

Better coders do not automatically become better collaborators

The benchmark evaluates ten open-weight models under five protocols: Single, Serial, Async-Private, Async-Manager-RO, and Async-Manager.

Multi-agent execution sometimes helps conventional coding performance. Six of ten models improved Test Pass Rate under their best multi-agent protocol relative to Single, with gains ranging from 7.8% to 22.7%. Four models nevertheless remained worse than their Single-agent baseline.

More revealingly, improvements in coding capability and improvements in dependency resolution do not move together consistently.

Scaling Qwen3.5 from 9B to 27B improved Test Pass Rate under every protocol, but ADPR did not improve uniformly. Likewise, the newer Qwen3.8-27B substantially outperformed Qwen3.6-27B in standalone results, yet under Async-Manager their final ADPR values were much closer: 64.9% versus 63.2%.

Protocol choice behaves similarly. No collaborative protocol improved dependency resolution across every evaluated model.

The benchmark therefore supports treating collaboration as a system-level capability. Model checkpoint, decomposition strategy, concurrency, managerial feedback, and repair permissions interact. Orchestration is not merely a wrapper whose effect can be assumed once the strongest coding model has been selected.

Dependency closure often arrives in concentrated phases

The trajectory measurements add another diagnostic.

Among collaborative protocols, 34.9% to 67.8% of eventual dependency closures fell within the representative normalized interval $0.50 < x \leq 0.75$. Resolution therefore often accumulated in concentrated phases rather than smoothly throughout execution.

This should not be turned into a universal “75% rule.” The paper explicitly reports substantial variation across models and protocols, and checkpoint normalization sharpens the aggregate peak. A raw-checkpoint analysis nevertheless finds checkpoint $k=2$ to be the most common first-closure point across all four collaborative protocols, suggesting that the concentration is not purely a normalization artifact.

For an operator, DRS-like monitoring is consequently useful less as a latency KPI than as a workflow diagnostic. It can distinguish a run whose critical interfaces close early from one in which integration depends on a late coordination burst—or never arrives at all.

The quality metric can change which protocol looks cost-efficient

Token cost alone does not settle whether additional coordination is worth paying for. The answer also depends on what the organization calls quality.

The paper constructs within-model Pareto frontiers using identical protocol token costs but alternates the quality measure between Test Pass Rate and ADPR. For five of the ten models, frontier membership changes.

That result affects an actual purchasing and architecture decision. A protocol that appears inefficient when judged by general repository tests may remain attractive when the objective is reliable cross-component integration. The reverse can also occur.

Cognaptus therefore infers that protocol comparisons should report at least two quality dimensions when component handoffs matter: overall task correctness and dependency satisfaction. Otherwise, a cost optimization can silently optimize the wrong outcome.

Dependency checks can become an engineering control

The paper directly shows that explicit dependency measurement reveals failures hidden by aggregate test performance. It also shows that model strength and coordination protocol do not provide reliable substitutes for measuring those dependencies.

For deployment, several practices follow as reasonable extensions of that evidence:

• Release acceptance: retain repository-level tests, but add targeted dependency checks for interfaces whose failure would invalidate the integrated change.
• Orchestration evaluation: benchmark protocol and model together rather than declaring one asynchronous or manager-based architecture universally superior.
• Permission design: compare read-only managerial coordination with bounded repair authority where direct intervention carries additional risk.
• Runtime diagnosis: track when critical dependencies first become satisfied across meaningful integration checkpoints.
• Cost governance: evaluate token spending against collaboration-aware quality as well as conventional test success.

These are business-use inferences from the benchmark, not effects directly measured in production engineering organizations.

The benchmark identifies a measurement problem, not a universal architecture

External validity remains limited. The benchmark contains 19 tasks from 16 repositories; 16 tasks are Python-only, while three combine Python and C++. It evaluates ten open-weight checkpoints rather than the full coding-agent landscape.

The protocol contrasts are carefully controlled system comparisons, but they are not randomized field experiments. DRS measures logical checkpoints rather than elapsed time or fixed compute. The reported concentration of dependency closure is also configuration-dependent rather than a general temporal law.

Those boundaries restrict claims about which architecture an organization should deploy. They do not weaken the narrower measurement result: passing repository tests and successfully resolving cross-agent dependencies are distinct properties, and either can be missed if only the other is measured.

For multi-agent software systems, evaluating the code is no longer enough. The handoffs need their own evidence.

Cognaptus: Automate the Present, Incubate the Future.