TL;DR for operators
A scene-by-scene assistant can describe every clip fluently while quietly forgetting who the characters are, how they relate, or why an earlier event matters now. StoryTeller addresses that continuity problem without task-specific training by keeping a persistent record of recurring characters and carrying forward only narrative facts that have been checked against the video.1
The advantage is clearest when understanding the current scene requires earlier context. In StoryAD-QA’s most context-dependent setting, StoryTeller scores 0.788 versus 0.673 for AutoAD-Zero—an 11.5-percentage-point gain. It does not lead every lexical-overlap metric, but it performs better when evaluation asks whether the narration preserves enough information to understand the story.
The operational choice is therefore not only which larger model to buy. Teams building long-form multimodal systems should also consider verified external state, persistent character tracking, and task-based evaluation that tests whether identities, relationships, and unresolved events survive across clips.
A fluent clip can still break the story
A scene-by-scene system may produce competent descriptions while quietly changing a character’s identity, dropping a relationship established ten minutes earlier, or describing an action without the event that made it consequential. The defect often appears later, when a listener can no longer reconstruct the story.
This creates an allocation decision. A team can spend more on a stronger backbone and a larger context window, or it can build explicit state around the model. StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description tests the second approach. Its results do not show that model quality is irrelevant. They show that model quality alone does not supply reliable continuity across a feature-length sequence.
Continuity is stored outside the model
StoryTeller processes a movie chronologically and carries two persistent structures from clip to clip.
The first links recurring face appearances to the same character. Face detections are grouped into temporal tracklets, encoded, and associated with dynamically updated identity nodes. Later clips can therefore use accumulated identity evidence instead of inferring every name again from local pixels and prompt history.
The second stores visually verified narrative facts. The system proposes structured facts from the current scene, prior accepted facts, and optional IMDb passages. A low-cost semantic filter selects proposals for a more expensive video-language-model check. Only candidates accepted against the current video can update memory or narration.
This sequence is the architecture’s main governance control. Retrieved text may suggest that a person is a named character or that an event has particular significance. It cannot become durable state merely because it fits the existing narrative.
The memory applies a reinforcement-and-decay rule:
Every stored fact loses weight as the movie advances, while facts related to the current scene receive reinforcement. Recurring characters, relationships, and unresolved events tend to remain available; isolated details recede. Moderate changes to reinforcement and decay keep StoryAD-QA accuracy within three percentage points and CIDEr within six percentage points of the default, indicating that the result is not tied to one narrow parameter setting.
Metadata proposes; video decides
Public movie metadata creates an obvious concern: perhaps the system is recovering the plot from IMDb instead of grounding descriptions in the film.
The architecture restricts that route. IMDb passages generate candidate names and context, but semantic similarity is only a screening step. A dedicated verifier must accept each candidate against the current clip before it enters persistent memory. The semantic gate is reported to be roughly 3,000 times faster than full visual verification, making it a cost filter rather than a factuality judgment.
Metadata still contributes materially. Removing IMDb lowers MAD-Eval scores from 21.4 to 17.2 CIDEr, 6.7 to 4.8 SPICE, and 15.3 to 12.2 ROUGE-L under the fixed Qwen3-VL setup. Yet the no-IMDb system retains identity tracking, structured verification, and memory, and remains competitive with or above AutoAD-Zero across the reported StoryAD-QA settings.
The transferable control is therefore not simply “add retrieval.” Retrieved material may nominate a fact; primary evidence controls admission into durable memory.
The gains grow when prior context becomes necessary
StoryAD-QA tests information preservation from generated audio-description text rather than direct video question answering. The answering model receives the descriptions, a question, and shuffled choices. It does not receive the video, title, metadata, or dialogue.
Track A tests questions answerable from descriptions within segments ranging from 30 to 240 seconds. Track B is more demanding: questions depend on preceding narrative context plus a fixed 30-second target clip, while the answerer receives only the target clip’s descriptions. Earlier facts must therefore be carried forward into the current narration.
| Evaluation | StoryTeller | AutoAD-Zero | Interpretation |
|---|---|---|---|
| Track A, 30 seconds | 0.892 | 0.840 | Better local information preservation |
| Track A, 240 seconds | 0.972 | 0.917 | Advantage remains over longer segments |
| Track B, 30+30 | 0.754 | 0.662 | Earlier context is surfaced in the target description |
| Track B, 90+30 | 0.788 | 0.673 | Largest gain, at 11.5 percentage points |
The complete system performs best in the longest and most context-dependent conditions, although simplified variants sometimes score higher in short or local cells. That pattern is more consistent with a continuity mechanism than with a universal improvement in sentence generation.
The ablations point to a system, not one decisive module
Under a fixed Qwen3-VL backbone and common MAD-Eval split, removing any major component lowers CIDEr, SPICE, and ROUGE-L. The VLM-only variant falls to 11.4 CIDEr, 4.3 SPICE, and 9.6 ROUGE-L, compared with 21.4, 6.7, and 15.3 for the full system.
Identity removal produces relative declines of 16.4% in CIDEr, 16.4% in SPICE, and 19.6% in ROUGE-L. Removing memory lowers the scores to 18.0, 6.0, and 13.2. Removing the structured fact schema reduces them to 15.4, 4.4, and 12.5.
These are controlled component tests, not causal estimates beyond the implemented pipeline. Within that pipeline, they weaken the assumption that a longer prompt or stronger backbone can substitute for explicit identity, fact structure, and memory management.
Caption overlap is not story comprehension
On MAD-Eval, StoryTeller is competitive but not uniformly superior. With VideoLLaMA2, it records 19.1 CIDEr against AutoAD-Zero’s 22.4, while scoring higher on SPICE, 9.0 versus 7.3, and ROUGE-L, 16.0 versus 14.4. The Qwen3-VL version reaches 21.4 CIDEr, 6.7 SPICE, and 15.3 ROUGE-L.
A deployment review based only on the highest overlap score could therefore reject the system that preserves more usable narrative context. StoryAD-QA asks a different question: can an answerer recover story information from the descriptions alone?
Human preference adds another evaluation layer. In a blind randomized comparison covering 50 clips and three evaluators, StoryTeller is preferred in 76.0% of judgments, compared with 5.8% for AutoAD-Zero. Ties and “both bad” each account for 9.1%.
The human study is small, so it cannot establish broad audience preference. Its value comes from complementing lexical resemblance and task-based comprehension rather than replacing either measure.
What product teams can carry into other long-horizon systems
What the paper directly shows: In the reported movie benchmarks, a training-free pipeline with persistent identity, verified structured facts, and relevance-weighted memory improves narrative QA and human preference relative to a strong training-free baseline. Component removals reduce caption metrics, and moderate memory-parameter changes do not erase the result.
What Cognaptus infers: Long-horizon multimodal assistants should externalize state when later outputs depend on stable entities and prior events. The affected user is the listener, reviewer, or downstream agent. The decision is what information enters persistent context. The relevant condition is a sequence where local observations are insufficient. The boundary is that retrieved or previously stored context must not be treated as fresh evidence without verification.
For media-accessibility operations, this supports generating first-pass descriptions for titles lacking licensed training corpora, then routing them to human editors. Evaluation should include questions testing identity, relationships, and causal continuity, not only similarity to professional reference wording.
For other multimodal systems, the reusable pattern is identity-linked state, structured candidate facts, evidence-based admission, and relevance-based retention. It fits environments where forgetting creates downstream inconsistency and proposed facts can be checked against a primary source.
The current deployment boundary is narrow
The main evaluation covers 10 MAD-Eval movies. Generalization across genres, languages, editing styles, and non-movie video remains unresolved. StoryAD-QA construction and answering depend on Gemini models, with some API-blocked inputs excluded from aggregate results. The paper reports 2,574 questions, while the current cleaned repository release contains 2,572, so benchmark versions require explicit tracking.
Persistent state can also preserve mistakes. Incorrect face-tracklet associations may attach the wrong name to a character, and repeated use across several clips can temporarily amplify the error. Verification limits unsupported metadata entry, but it does not eliminate identity failures originating in visual association.
Cost is another constraint. A feature-length movie requires roughly three to four hours on one A100-SXM4-80GB GPU. This is compatible with offline draft production; it does not establish real-time or low-cost catalog-scale operation.
Long-form coherence is an architecture decision
StoryTeller’s clearest contribution is not a claim that movie audio description has been solved. It provides a more testable account of where long-form coherence comes from. The model observes the current clip. External state keeps identities and prior facts available. Verification controls what becomes persistent. Reinforcement and decay determine what remains active. Task-based QA checks whether the resulting narration still carries enough information to understand the story.
For teams choosing between a larger model and a better state architecture, the paper shifts the default. Increase backbone capability where perception is weak. Build explicit, verified memory where continuity is the failure.
Cognaptus: Automate the Present, Incubate the Future.
-
Seung Hyun Hahm and Minh T. Dinh and SouYoung Jin (2026). StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description. arXiv:2607.11798. https://arxiv.org/abs/2607.11798 ↩︎