Who Did What, When—and From Which Camera? The Perception Gap Behind Video Agents

TL;DR for operators

A model can correctly recognize what is visible in a video and still lose track of who performed an action, how many times an event occurred, or which viewpoint observed it. GameplayQA¹ makes those failures separately measurable.

Across 16 evaluated multimodal models, average accuracy falls from 61.2% on single-reference tasks to 56.0% on temporal tasks and 49.4% on synchronized cross-video tasks. Occurrence counting averages just 36.5%, while cross-video ordering reaches 38.8%. Other-agent actions and states are also harder than questions about world objects.

For model selection, the practical implication is narrower than “video models are not ready.” The paper suggests that aggregate video-QA accuracy can conceal operationally different failure modes. A system expected to operate across changing scenes or multiple cameras should be tested separately for temporal grounding, agent attribution, and viewpoint alignment.

The boundary is equally important: GameplayQA evaluates perception and reasoning over observed video. It does not evaluate whether an agent selects the correct next action, estimates action value, or succeeds in closed-loop control.

Recognition is only the first part of maintaining a scene

Consider a system that can describe a frame accurately. It recognizes a vehicle, another character, an object on the ground, and an action taking place. That competence is useful, but an autonomous system faces a harder requirement once the scene starts changing: it must preserve the relationships among observations.

Who performed the action? Did it happen before or after another event? Was the same action repeated three times or four? Did two cameras observe the same event from different positions, or different events that merely look similar?

GameplayQA turns that distinction into a benchmark architecture. Its main contribution is not simply another collection of video questions. The framework decomposes observations into information about the observing agent itself, other agents, and the surrounding environment. Only after establishing those separate tracking requirements does the paper formalize them as Self–Other–World, represented through six annotation tracks covering actions and states for self and others, plus world objects and events.

The benchmark contains 2,365 multiple-choice questions from 100 videos across nine commercial 3D games. Questions progress from single-reference recognition to temporal reasoning and then to reasoning across synchronized viewpoints.

That progression exposes a consistent deterioration. Models average 61.2% at the first level, 56.0% at the temporal level, and 49.4% when multiple synchronized videos must be reconciled. The best reported model overall, Gemini 2.5 Pro, reaches 71.3%, compared with reported human accuracy of 80.5%.

The result is not evidence that recognition is unimportant. It shows that recognition accuracy and maintenance of an evolving scene representation should not be treated as interchangeable capabilities.

The failures concentrate around time, identity, and perspective

The difficult questions are not distributed uniformly.

Occurrence Count, which asks how many times an action or event happened, produces average model accuracy of 36.5%. Cross-Video Ordering, which requires events from multiple synchronized videos to be placed in the correct temporal order, averages 38.8%.

Agent identity introduces another gap. Aggregate accuracy is 54.0% for Other-Action and 55.4% for Other-State, compared with 62.0% for World-Object questions. The paper also reports that temporal and cross-video distractors generate more errors than scene distractors.

Taken together, these diagnostics point toward a binding problem: models must associate the right observation with the right time, entity, and perspective while the underlying scene continues to change.

This interpretation is reinforced by the benchmark’s treatment of environmental speed. The authors define decision density as the frequency of semantically relevant annotated information:

GameplayQA contains 2,709 verified true labels across 2,219.41 seconds, or roughly 1.22 labels per second. Faster-paced games with rapid state transitions are generally harder, and errors tend to increase with longer clips and additional synchronized perspectives.

The terminology needs a boundary. These labels cover actions, states, objects, and events. The metric does not count actual policy decisions, estimate policy entropy, or measure whether a model chose the optimal action. It describes how rapidly the annotated information stream changes.

The ablation shows that the benchmark is using video, not just question priors

A reasonable concern with multiple-choice video benchmarks is that models may answer from linguistic cues or static visual recognition without tracking the sequence itself.

The paper addresses that concern with a degraded-input ablation on GPT-5-mini. Full-video performance is 62.7%. Removing the video reduces accuracy to 29.4%; supplying one random frame reaches 41.7%; shuffling the frames reaches 54.8%.

The shuffled condition is particularly informative. Its effect is relatively small on single-reference questions but larger on temporal and cross-video questions. That is consistent with the benchmark requiring temporal order rather than merely access to representative imagery.

It does not establish the internal mechanism by which a model reasons. Structured distractors and degraded inputs diagnose observable error patterns; they do not reveal a model’s causal internal process.

The reusable contribution is diagnostic evaluation, not a single leaderboard

For organizations evaluating multimodal backbones, Cognaptus sees the paper’s most transferable contribution in the benchmark construction process.

The framework annotates timelines densely, separates self, other agents, and world state, creates distractors designed around specific confusion types, and generates questions that isolate different reasoning demands. That permits an evaluator to distinguish, for example, failure to recognize an object from failure to associate an observed action with the correct person or camera.

This changes the model-selection question for several operational users. A robotics team evaluating perception for interaction with people may care disproportionately about other-agent attribution. An autonomous-vehicle system operating over video streams may need temporal localization and recurrence tracking. Multi-camera analytics may require cross-view identity and ordering consistency. In each case, an aggregate benchmark score can average away the failure mode most relevant to deployment.

The implication is not that GameplayQA scores should become a deployment gate for those domains. Rather, its evaluation architecture provides a template for constructing domain-specific tests around the bindings a system must preserve.

Real-world transfer is promising evidence for the method, not validation of deployment readiness

The authors apply the same pipeline to dashcam collision footage and synchronized videos of humans collaboratively assembling Lego. This exploratory transfer test contains 213 questions from four videos, totaling about 113 seconds at 0.50 labels per second.

The broad difficulty pattern remains similar: Gemini 2.5 Pro leads, while occurrence counting and cross-video ordering remain difficult. That supports the narrower proposition that the benchmark-building approach can be applied outside games.

Four videos cannot establish broad generalization to autonomous driving, industrial robotics, surveillance, or other real operational environments. The transfer result is evidence about portability of the evaluation framework, not proof that gameplay performance predicts deployment performance.

There are additional construction costs. Dense annotation is labor-intensive: the paper reports 25–35 minutes to annotate a 30-second clip. Because verified labels are reused to generate multiple questions, an error in timing, entity type, or content can also propagate. Human quality checks reduce this risk but do not remove it; about 8% of human-evaluated questions were flagged as faulty or ambiguous.

Evaluate the bindings your system cannot afford to lose

GameplayQA narrows an important question in multimodal AI evaluation. A model may recognize objects and actions competently while remaining substantially less reliable when it must maintain a temporally ordered, agent-specific, multi-perspective representation of an unfolding scene.

For operational evaluation, that suggests separating at least three questions: does the model know what happened, can it preserve when and how often it happened, and can it maintain who and which viewpoint each observation belongs to?

Those tests still stop before control. Perception benchmarks cannot tell an operator whether a system will choose the right action once its scene representation is formed. But they can reveal whether the information supplied to that decision process is already losing its temporal, entity, or perspective grounding.

That is a more actionable use of video benchmarking than treating one aggregate accuracy number as a proxy for agent readiness.

Cognaptus: Automate the Present, Incubate the Future.