Hear First, Commit Later: Voice-Light's Turn-Taking Tradeoff

TL;DR for operators

A conversational voice assistant must decide when to respond, when to stop speaking, and which parts of its generated response actually reached the user. These decisions have different costs and should not depend on a single turn-taking prediction.

Voice-Light demonstrates a hybrid architecture that separates uncertain evidence from consequential actions. Acoustic onset can trigger reversible playback ducking, while stronger evidence governs interruption and cancellation. Language and speech generation can begin privately before the user’s turn is confirmed complete, and only browser-acknowledged playback enters durable conversation history.

The central benchmark is negative: a historical learned endpointing policy achieved 12.53% end-of-turn recall versus 95.60% for Silero, with both producing 2.70% false cutoffs. Yet the deployed hybrid agent recorded a 758 ms median from final voice-activity endpoint to first server audio across 36 operator-run turns.

For deployment teams, the lesson is to preserve inexpensive timing baselines, isolate speculative work, verify delivered speech, and measure endpoint accuracy, audible latency, and interruption quality separately. The paper establishes a working engineering approach, not proven conversational superiority.

A short pause creates several different decisions

Imagine a customer telling a voice assistant to change a booking. The customer pauses to recall a date, then continues. If the assistant responds during that hesitation, it may answer an incomplete request.

Later, the assistant is explaining the booking options when the customer says, “Right.” Should the assistant stop, briefly lower its volume, or continue? A short acknowledgment and a genuine interruption can initially sound similar, although their consequences are different.

These situations reveal a limitation of treating conversational responsiveness as a single prediction about whether someone has finished speaking.

Bertil Braun’s Voice-Light¹ addresses this problem through a full-duplex cascaded architecture. Speech recognition, language generation, tools, speech synthesis, and playback remain separately controllable.

Its most consequential result is not that a learned turn-taking model solves conversational timing. In fact, the historical learned policy performed substantially worse than a simple acoustic timing baseline. The system’s contribution is showing how a functioning voice agent can retain useful learned signals without depending on their superiority.

The learned endpointing policy lost to Silero

End-of-turn detection decides whether observed silence indicates that a person has finished speaking or is temporarily holding the conversational floor.

A premature decision can interrupt the user. An excessively delayed decision makes the assistant feel unresponsive. The relevant operational comparison therefore considers both false cutoffs and successful recognition of completed turns.

Voice-Light evaluates a historical learned completion policy against deployable alternatives on a locked V1 real-conversation test.

Source: Section 6.3, Table 4. The Voice-Light result belongs to the historical step-3,500 checkpoint, not the deployed step-750 adapter.

At the same observed false-cutoff rate, Silero identified completed turns far more reliably and committed faster on average. The learned policy did not earn the authority to replace the timing baseline.

The test contained 1,673 silence candidates from 11 conversations, including 37 HOLD cases and 1,636 end-of-turn cases. Each policy made one HOLD error, so the shared 2.70% false-cutoff rate rests on a small denominator. The policies also received acoustic evidence at different native timing gates. These results compare complete operational policies, not isolated detectors exposed to identical information.

This was the main locked-test comparison, not an ablation or a demonstration that all learned endpointing approaches are inferior.

A separate V2 semantic-completion evaluation remained validation-only: no candidate satisfied the predefined deployment gate, so the locked V2 test was not opened.

For a voice-product team considering a learned endpointing replacement, this evidence supports a concrete procurement requirement: the candidate must outperform the incumbent timing policy on held-out natural conversations under comparable operational constraints.

Reusing speech representations did not solve natural turn prediction

Voice-Light attempts to obtain conversational signals economically from an existing speech-recognition system.

Its persistent Nemotron Speech Streaming 0.6B encoder supplies frozen intermediate representations from layers 6, 12, 18, and 24. A roughly 183,000-parameter adapter combines these features with assistant-speaking state through causal convolutions and a unidirectional recurrent network.

The adapter estimates completion and interaction-related signals without requiring another complete speech backbone. This demonstrates architectural reuse, although the paper does not establish isolated efficiency superiority over alternative turn-taking models.

Training reveals the harder problem: transferring synthetic interaction patterns to actual conversations.

The large synthetic-to-human decline suggests that generated speech timelines did not adequately capture the distinctions needed for natural conversational completion. Human fine-tuning improved discrimination only modestly.

The deployed step-750 adapter achieved 65.04% end-of-turn recall and 3.73% false cutoffs under its selected human-validation policy, but these are not locked-test measurements and cannot be substituted for the historical checkpoint’s results.

The implication is specific: synthetic interaction data may supply inexpensive supervision, but learned turn-taking signals require independent natural-conversation validation before gaining control over consequential actions.

The controller reacts before it commits

Voice-Light’s hybrid controller assigns different levels of authority to different evidence.

When Silero detects speech onset while the assistant is talking, the system can immediately lower playback volume or pause. This response is reversible. If the sound turns out to be a brief acknowledgment, playback can resume.

Committed interruption requires stronger evidence, combining learned interaction signals, transcript information, and bounded decision timeouts.

The sequence separates three decisions:

1. Detect overlap: react quickly to acoustic evidence.
2. Interpret the interaction: distinguish a possible interruption from non-floor-taking feedback.
3. Commit cancellation: terminate the assistant’s current response only when the controller has sufficient grounds.

This avoids forcing an uncertain acoustic observation to authorize an irreversible action immediately.

It is an implemented control mechanism, not a demonstrated guarantee of interruption accuracy. The paper does not independently establish the natural-conversation calibration of its floor-taking and backchannel predictions.

The business application is clearest in customer-facing voice workflows where accidental cancellation can discard an explanation or disrupt a multi-step transaction. Such systems can use reversible reactions to improve responsiveness while reserving committed decisions for better-supported evidence.

Speculative speech stays private until it is valid

Waiting for definite turn completion before starting every downstream operation creates avoidable sequential delays.

Voice-Light instead begins possible language-model and speech-synthesis work before final turn commitment. Candidate text and audio remain in private buffers.

If the recognized transcript remains sufficiently stable and the candidate is still valid, the controller can promote the prepared response. If the user continues speaking or the transcript changes materially, the system can discard the candidate.

Generation identifiers and cancellation barriers prevent obsolete outputs from reappearing after a newer interaction has superseded them.

A second boundary concerns memory.

Generated speech is not necessarily heard speech. A language model may produce several sentences, while the user interrupts playback after the first.

Voice-Light uses browser playback acknowledgments and audio positions to determine what was actually rendered. Only acknowledged speech contributes to durable assistant history.

This matters when subsequent reasoning depends on what the user has been told. Recording an unheard explanation as completed dialogue could cause the assistant to rely on information that never reached the customer.

The architecture therefore separates internal preparation, authorized output, and durable conversational state.

Its typed tool execution follows a similar separation. Synthetic spoken-tool fine-tuning improved Qwen3-1.7B’s correct tool decisions from 80.0% to 95.2% on a shared-generator holdout. That experiment demonstrates protocol learning, not general tool competence. Production uses a different model, Qwen3-4B Instruct.

The deployment was responsive, but speculation’s benefit remains unproven

The integrated system was tested during three unscripted browser microphone sessions conducted by one operator.

Across 36 measured response turns, median latency from the final voice-activity-detection endpoint to first server audio was 758 ms. Mean latency was 932 ms, and 21 turns were below 800 ms.

A fully exported 13-turn trace also reported an 836 ms median from final VAD endpoint to browser rendering. The component intervals overlap and cannot simply be added together.

Within that traced subset, nine turns with successfully promoted speculative work had a median first-server-audio latency of 667 ms, compared with 1,513 ms for four non-promoted turns.

This comparison is observational. Promotion depended partly on transcript stability and turn difficulty, so the 846 ms difference is not a causal estimate of speculation’s speedup.

Deployment introduces another constraint: the final two-GPU, scale-to-zero configuration experienced approximately 32–55 seconds of cold-start readiness. The warm-response latency results exclude that delay.

For intermittent voice services, operators must therefore evaluate the cost of keeping capacity ready against the startup experience users will encounter.

What an operational evaluation still needs

Voice-Light supplies a reproducible architecture and instrumented engineering observations, but several acceptance questions remain unresolved.

The locked endpointing test covers few independent conversations and HOLD cases. The deployed adapter is not the historical checkpoint evaluated in that test. The latency case study has no representative participant sample, controlled acoustic environments, or independently established conversational-quality outcomes.

These boundaries suggest distinct evaluation requirements for a production voice assistant:

• Endpointing: Compare locked natural-conversation performance with inexpensive timing baselines.
• Interruption handling: Test acknowledgments, genuine interruptions, hesitation, echo, and overlapping speech with independent labels.
• State integrity: Verify that canceled or unheard generated speech cannot influence later dialogue.
• Responsiveness: Measure both first server audio and actual browser playback, including cold starts and external tool delays.
• Generalization: Repeat evaluations across users, accents, noise conditions, and network environments.

These tests address different failure modes. Strong performance on one does not establish reliability on the others.

The architecture earns more confidence than the learned detector

Voice-Light demonstrates that a cascaded voice agent can combine acoustic detection, learned conversational features, speculative processing, typed tools, and playback-aware history while preserving explicit control over consequential actions.

The historical learned completion model did not outperform a simple timing baseline. That failure materially shaped the deployed system: rather than granting learned predictions exclusive control, the architecture retained reliable timing mechanisms and limited uncertain signals to roles appropriate for their evidence.

For organizations building voice assistants, the more transferable contribution is this allocation of decision authority. Act quickly when the action is reversible; demand stronger evidence when it changes the conversation irreversibly.

Voice-Light establishes that such a design can be implemented and instrumented. Whether it delivers consistently better human conversations remains a separate empirical question.

Cognaptus: Automate the Present, Incubate the Future.