The Voice Layer Can Become the Bottleneck

TL;DR for operators

A realtime agent has at least two different jobs: participate in a conversation at the right moment, and complete whatever task the conversation initiates. Gander separates those jobs. A 9B multimodal front layer continuously handles audio, video, turn-taking, and interruption, while a separate asynchronous back brain can take over longer reasoning or tool workflows.

The reported results show why that separation is operationally meaningful. Gander achieved a 100% take-turn rate and an 8.0% premature-interruption rate on Full-Duplex-Bench v3, yet its end-to-end Pass@1 was only 0.400. When the same back-brain agent was driven directly from user text, ToolSel increased from 0.759 to 0.934, RespQual from 0.490 to 0.740, and Pass@1 from 0.400 to 0.520.

For product teams, this changes where debugging effort should go. A stronger reasoning backend does not automatically produce a stronger realtime agent. Routing, speech mediation, delegation, task state, and interaction control can become capability bottlenecks of their own.

The unresolved part is duration. The paper evaluates interaction, multimodal understanding, and agent execution across several benchmarks, but does not yet provide a comprehensive test of sustained Brain-Cerebellum collaboration over long-running tasks.

A voice agent has to succeed on two clocks

A service agent cannot simply wait for a clean user turn, process a request, and return an answer. In a live conversation it must decide when someone is still speaking, when a response is expected, whether an interruption is appropriate, and how to keep the interaction moving while a longer task continues elsewhere.

Those demands create a problem that backend model capability alone does not resolve. A system can be good at deciding when to participate and still be much weaker at completing the task being discussed.

Gander, introduced in Multimodal Duplex Interaction Agent¹, makes that distinction unusually visible. On Full-Duplex-Bench v3, it took the conversational floor appropriately in all 100 reported scenarios and recorded an 8.0% premature-interruption rate, the lowest among the systems in the table. Its end-to-end Pass@1, however, was 0.400, below several reported realtime baselines.

That result is more informative than a single aggregate ranking. Conversational timing and task competence are separable system problems.

Gander assigns the two problems to different layers

The paper’s main architectural decision is to divide work across two computational timescales.

The front “cerebellum” is a 9B model adapted from MiniCPM-o 4.5. It continuously processes streaming audio and visual input. Rather than treating turn-taking as an external speech-boundary rule, Gander predicts an interaction action—listen, speak, or interrupt—inside each one-second chunk before generating response content.

Audio tokens, visual tokens, the interaction-control decision, and generated text are flattened into a single causal stream. The model maintains a rolling context of 128 chunks, roughly two minutes of interaction.

Longer work is handled differently. Complex or sustained tasks can be delegated through structured task_start, task_send, and task_resolve operations to an asynchronous back brain. An orchestration runtime tracks persistent projects, tasks, execution runs, worker events, permissions, and deliveries while the front layer remains available to the user.

This architecture should not be read as one larger model divided into conceptual modules. The back brain is a separate general-purpose agent. The design choice is therefore less about giving one model more internal capacity than about deciding which responsibilities belong in the low-latency interaction loop and which can leave it.

The strongest diagnostic result is the capability lost in integration

The paper’s back-brain-only comparison is especially useful because it tests what happens when the realtime front layer and audio path are bypassed.

The direct-text condition is not a competing voice-agent score. It has no turn-taking measurements and bypasses the front cerebellum and speech channel. Its purpose is diagnostic.

The gap shows that backend capability does not pass through the integrated system unchanged. The paper does not isolate how much loss comes from delegation decisions, speech recognition, synthesis, routing, or other parts of the interaction path. It does show that these components collectively matter enough to reduce several task metrics substantially.

For an operator, that changes the debugging hierarchy. If an agent’s backend performs well in direct execution but the deployed realtime product does not, replacing the reasoning model may address the wrong layer.

Native multimodal input helps, but interaction training is not free

Gander also provides evidence that continuous audio and video should not necessarily be reduced to separate upstream signals.

On WorldSense, joint audio-visual input reached 49.62%, compared with 44.61% for video only and 43.32% for audio only. On Daily-Omni, joint input reached 78.53%, versus 59.40% and 57.81%. The paper defines the resulting fusion gains as 5.01 and 19.13 percentage points respectively.

That ablation supports the value of combining modalities inside the interaction model when the answer depends on temporally aligned visual and acoustic information.

There is a separate trade-off. Gander was initialized from MiniCPM-o 4.5, yet after interaction-oriented fine-tuning it scored 6.08 points lower than that base model on WorldSense and 1.67 points lower on Daily-Omni. Interaction specialization therefore did not produce uniform improvement in general multimodal understanding.

The useful distinction is between fusion and retention. Joint audio-video input improves Gander relative to its own unimodal variants; that does not mean the interaction-trained model exceeds its initialization on the same multimodal benchmarks.

For product teams, orchestration becomes part of model quality

Paper evidence: Gander demonstrates that a realtime multimodal front end can continue managing conversation while longer work is delegated asynchronously. It also shows that the integrated path performs materially worse than direct access to the same back-brain agent on several tool-task metrics.

Cognaptus inference: For customer support, field assistance, live-screen guidance, and similar systems, it may be useful to treat the interaction layer and execution layer as separately optimizable components. The front layer can be tuned for latency, interruption, multimodal grounding, and conversational control, while the back end can be changed as reasoning or tool capabilities improve.

That separation also creates an engineering obligation. Task creation, cancellation, permission handling, state synchronization, routing, and result delivery are no longer peripheral middleware. They determine how much of the backend agent’s capability survives deployment.

The replaceable-backend advantage is therefore conditional: modularity helps only when the interface between modules preserves enough task intent, context, and execution state.

The missing benchmark is sustained collaboration

The paper provides substantial comparative evidence for turn-taking, spoken question answering, multimodal understanding, modality fusion, and tool-assisted interaction. Gander also leads the reported full-duplex models on both SpokenQA subsets, showing that realtime interaction constraints do not prevent strong short-form factual performance.

What remains less established is the behavior the architecture is ultimately designed to support: extended collaboration in which conversation continues while a back brain performs a long-running task, encounters intermediate states, requests permissions, returns partial results, and resumes work over time.

The authors identify related open problems in memory, long-context management, richer Cerebellum-Brain communication, reward design, credit assignment, and optimization for long-horizon duplex interaction. Existing benchmarks largely evaluate the participating capabilities separately.

That limits the operational conclusion. Gander demonstrates a credible architecture for separating fast interaction from slower execution and provides useful diagnostics of where the stack currently loses capability. It does not yet establish how reliably that architecture performs when both layers must collaborate continuously over a long task.

The architecture changes what should be optimized

Gander’s most instructive result is not that a realtime agent can speak naturally while calling tools. It is that the system performs very differently depending on which layer is being measured.

The front cerebellum is strongest where realtime interaction demands specialized behavior: deciding when to listen, speak, and interrupt. The back brain is stronger when evaluated without the realtime interaction path. Joint audio-video perception adds information that neither modality supplies as effectively alone.

For teams building realtime agents, those results argue for measuring the layers separately before optimizing the system as a whole. Backend reasoning, interaction control, multimodal perception, and orchestration are connected, but they are not interchangeable sources of capability.

The engineering problem is therefore not merely to attach speech and tools to the strongest available model. It is to preserve capability as work crosses between fast conversational control and slower execution.

Cognaptus: Automate the Present, Incubate the Future.