Correct on the Frame, Wrong on the Timeline

TL;DR for operators

A video system can produce the right answer without reliably understanding what happened over time. That becomes a procurement and QA problem when the decision depends on whether one event happened before another, how long something lasted, which direction it moved, whether an action repeated, or whether one event depended on another.

TimeBlind¹ tests this failure mode by making isolated correct answers insufficient. It pairs near-identical videos that differ in a targeted temporal property and asks complementary questions whose correct answers reverse across the pair. GPT-5 scores 77.3% on ordinary per-question accuracy but only 46.3% when required to get the entire paired instance right. Gemini 3 Pro shows a similar gap: 76.2% versus 48.2%. Humans reach 98.2% on the stricter measure.

For teams evaluating video AI, the implication is concrete: include controlled temporal contrasts in validation suites and score consistency across those contrasts. The paper also shows why adding model parameters, frames, sampling density, or reasoning tokens should not be treated as a substitute for this testing. Those interventions help in some settings, but none closes the diagnostic gap.

A correct answer can hide an inconsistent model

Ordinary video-question benchmarks typically score each question independently. That is reasonable when the product requirement is also independent question answering. It becomes less informative when a system is supposed to distinguish two situations that look almost identical but unfold differently.

TimeBlind changes the unit of success. Each of its 600 instances contains two minimally different videos and two complementary questions, producing four video-question trials. Instance Accuracy, or I-Acc, counts the instance as correct only when the model answers all four correctly.

This explains why the headline numbers fall so sharply. GPT-5 moves from 77.3% standard accuracy to 46.3% I-Acc. Gemini 3 Pro moves from 76.2% to 48.2%. The strongest evaluated model therefore fails the full consistency requirement on more than half of the benchmark instances, despite answering roughly three quarters of individual trials correctly.

That difference is the main measurement result. It does not show that ordinary accuracy is wrong; it shows that ordinary accuracy answers a weaker question. For applications where temporal contrasts change the required action, isolated correctness can overstate the reliability that matters.

Minimal pairs force temporal evidence to carry the answer

The benchmark is designed to reduce alternative routes to a correct response. The paired videos retain near-identical static content while changing temporal structure, and the complementary questions reverse the correct answer across the pair. A model cannot remain consistently correct merely by associating an object or scene with a plausible answer.

The paper then checks whether those protections actually work. These are validation tests of the benchmark design, rather than separate claims about model architecture.

With GPT-5, a single-frame condition reaches only 4.5% I-Acc. Language-only input reaches 1.5%. Shuffling the frames reaches 3.0%. Random instance-level performance is 6.3%. Humans, meanwhile, reach 98.2%.

Those results support a bounded conclusion: success on TimeBlind generally requires ordered temporal evidence, while several obvious static, linguistic, and order-insensitive shortcuts perform near or below the random instance baseline. They do not prove which internal representations a successful model uses.

Additional robustness checks point in the same direction. GPT-5 scores 45.6% on the highest-similarity video pairs and 47.8% on the lowest-similarity subset, making residual static dissimilarity an unlikely explanation for the overall difficulty. Model rankings are also stable between GPT-5-generated and human-written schemas, with no evidence in that comparison that GPT-5 benefits from answering schemas it generated.

More frames and more reasoning do not reliably buy temporal competence

A natural response to weak video reasoning is to increase resources. TimeBlind tests several versions of that strategy, and the pattern is mixed.

Scaling Qwen3-VL from 8B to 235B raises I-Acc from 19.7% to 25.8%. Increasing sampling density helps more at first: Gemini 3 Pro rises from 48.2% at 1 FPS to 56.2% at 5 FPS. But every tested model declines again at 10 FPS. GPT-5, for example, moves from 46.3% to 51.0% and then 49.3%.

More sampled frames show a similarly non-monotonic pattern. GPT-5 peaks at 50.8% with 16 frames and falls to 48.3% with 32.

Inference-time reasoning produces real gains, particularly for Qwen3-VL, but it does not remove the deficit. On the random 30% subset used for this computationally expensive ablation, Qwen3-VL-235B improves by 10.5 percentage points, from 25.6% to 36.1% I-Acc. GPT-5 improves 3.3 points, from 46.7% to 50.0%.

For infrastructure and model-selection teams, the relevant interpretation is narrower than “more compute does not work.” Additional compute can improve results. The paper shows that compute inputs and temporal reliability are not monotonic substitutes: a more expensive configuration still needs direct evaluation against the temporal distinction the product depends on.

The taxonomy tells teams what to regression-test

TimeBlind does more than produce one difficult leaderboard. It separates temporal compositionality into three levels: Events, Event Attributes, and Structural Event Logic.

The middle category is especially revealing. Event Attributes cover properties such as speed, force, magnitude, duration, direction, and repetition. Strong proprietary models perform worse here than on basic event recognition: GPT-5 reaches 32.4% I-Acc and Gemini 3 Pro 37.3%.

Structural Event Logic examines relationships among events, including temporal topology and causal contingency. Here the spread between model families is large. Gemini 3 Pro reaches 58.3% I-Acc, while Qwen3-VL-235B reaches 19.3%; the latter scores 7.5% on Causal Contingency.

Cognaptus infers a practical use from this decomposition. A robotics team whose failures depend on motion direction should not treat “video reasoning” as one regression target. A workflow system that must recognize whether step B followed step A needs different tests from a system estimating duration or repetition. The taxonomy gives teams a way to organize those targeted tests even when they do not adopt TimeBlind itself.

Use paired consistency in procurement and pre-deployment QA

For a model buyer, the useful change is in the acceptance test.

Start with cases where a temporal distinction would alter the downstream decision. Construct paired examples that preserve objects, scene, wording style, and other irrelevant cues while changing that temporal property. Then ask complementary questions across both cases and require consistency across the entire set, rather than averaging the individual answers.

This is particularly relevant when vendors or internal teams compare configurations using aggregate accuracy. A model that gains several points after higher FPS or additional reasoning may still fail the controlled contrasts that determine whether it can support a particular workflow. Paired tests convert that uncertainty into a directly observable failure rate.

The paper does not provide an ROI estimate for such testing, and it does not establish how TimeBlind scores map to production incident rates. The business case is instead diagnostic: stricter tests can reveal a class of temporal errors before aggregate benchmark scores are used as evidence for a deployment decision.

The evidence stops at short, controlled video reasoning

TimeBlind is deliberately narrow. Its videos average 8.49 seconds. Questions are in English and use binary or two-choice formats. Qualitative attributes such as force and magnitude retain some subjectivity despite strong annotation agreement. The benchmark also measures behavior; it does not identify causal mechanisms inside model internals.

Those boundaries matter when translating the results. The benchmark does not establish that a model scoring poorly will fail every real video workflow, nor that a high TimeBlind score would certify a system for production or safety-critical use. Long-horizon video understanding, multilingual use, open-ended generation, and deployment-specific reliability remain outside the demonstrated evidence.

Within its scope, however, the measurement problem is clear. If a workflow depends on time, evaluating answers one at a time can hide inconsistency across the very contrasts that determine whether the system understood the sequence. TimeBlind provides a stronger test: hold the scene nearly still, change what happened over time, and see whether the model changes its answer for the right reason.

Cognaptus: Automate the Present, Incubate the Future.