TL;DR for operators
An AI agent can interpret a video, search for physical specifications, and execute calculations without reliably combining those capabilities into a correct physical judgment.
TriPhysBench1 tests this coordination across 410 questions from 200 real-world videos. The strongest of six evaluated vision-language models achieves 50.38% answer correctness, measured as average partial credit rather than the percentage of exactly correct answers. Diagnostic scoring frequently flags failures to identify the quantities required for a solution.
The accompanying TriPhys training framework improves answer correctness by 16.4%–17.7% relative to three Qwen backbones on 110 questions from videos excluded from training. It learns complete reasoning and tool-use trajectories rather than only final answers.
For teams building video-grounded inspection assistants or physical-reasoning agents, the operational implication is to verify observations, retrieved specifications, mathematical derivations, and conclusions separately. The training results justify further evaluation of this approach, but do not establish deployment reliability or improved robotic execution.
A plausible physical estimate can conceal a missing measurement
Consider an inspection assistant asked to estimate a physical quantity from recorded equipment movement. It may need to identify the relevant event, estimate motion from video, retrieve a material or equipment specification, and apply the appropriate physical relation.
The calculation can be mathematically correct while its inputs are unsuitable. A visually estimated dimension might be inaccurate. A retrieved specification might describe a different material or operating condition. A final answer might use inconsistent units despite a valid intermediate derivation.
Each error affects a different part of the workflow, yet a conventional final-answer score may conceal where the problem occurred.
This is the coordination problem examined by TriPhysBench. Its central question is whether contemporary agents can connect physical observations, external knowledge, and quantitative reasoning under a common evaluation protocol.
Across six models given shared instructions, web search, Python execution, and up to six tool-loop turns, Gemini-3.6-Flash leads the full benchmark with an answer correctness score of 0.5038 (Section 4.2, Table 3).
That score does not mean exactly 50.38% of questions were solved. The metric awards partial credit under answer-type-specific rules. Nevertheless, the result indicates substantial difficulty even when the necessary tool categories are available.
Two ways an agent can be missing the physics
TriPhysBench separates physical questions according to the information an agent must acquire.
| Task | Missing information | Benchmark coverage |
|---|---|---|
| Measurement-grounded physical reasoning (MPR) | Observations or measurements recoverable from the video | 228 questions, 90 videos |
| Knowledge-grounded physical reasoning (KPR) | External physical properties or specifications | 182 questions, 110 videos |
MPR draws on skilled human activity recordings from EgoExo4D, while KPR uses physical-process recordings from BAAI-DataCube Physics-aware-videos. The benchmark represents events through graphs connecting participants, observations, physical properties, relations, and required quantities.
Difficulty is controlled by withholding designated inputs: zero to two video observations for MPR, and one to three external specifications for KPR. The 410 questions comprise 172 Easy, 134 Medium, and 104 Hard cases.
This construction makes information acquisition part of the evaluated task rather than automatically supplying every input needed for a calculation.
In the six-model comparison, GPT-5.5 achieves the highest MPR answer correctness, while Gemini-3.6-Flash leads KPR and the overall benchmark. Five of six models perform worse on MPR than KPR, although differences in question and answer-format composition prevent treating that gap as an isolated measure of perception capability.
All six models also show declining overall scores as information-withholding difficulty increases.
The operational lesson is that testing a formula solver with supplied inputs does not adequately assess an agent expected to discover those inputs itself.
The largest diagnostic problem appears before the equation
TriPhysBench evaluates more than final answers. Its solution correctness score (SCS) separates four dimensions:
- Quantity identification (25%): Were the necessary quantities and information sources identified?
- Formula selection (25%): Was an applicable physical relation selected?
- Derivation (30%): Were mathematical and numerical transformations valid?
- Conclusion (20%): Did the result follow from the derivation, units, and question?
Its overall benchmark score assigns 70% weight to answer correctness and 30% to solution correctness.
The diagnostic analysis reports quantity-identification failure indicators in approximately 56.0%–91.7% of the examined model-subtask combinations (Section 4.4, Figure 6; Appendix E.2). Conclusion-related failures are more frequent than derivation-related failures, while formula selection is comparatively less frequently flagged.
These are rubric-defined assessments of returned solutions, not independently observed causal failure mechanisms. Still, their distribution challenges an evaluation strategy focused mainly on arithmetic or equation selection.
For example, an assistant might select the correct energy equation but retrieve an inappropriate mass. Improving arithmetic would not resolve that input problem. Conversely, a valid derivation could still produce an unsupported conclusion if its assumptions or units are mishandled.
The study also checks solution-scoring consistency using a second language-model judge. GPT-5.5 and DeepSeek-V4-Flash produce a response-level SCS correlation of 0.877 and identical aggregate model rankings. This is a scoring robustness check; agreement between judges does not independently verify every physical quantity or reference calculation.
Training the complete tool-use sequence improves held-out scores
TriPhys investigates whether agents can learn better coordination from complete problem-solving trajectories.
Training begins with verified trajectories containing reasoning steps, tool calls, tool responses, and final answers. Supervised fine-tuning (SFT) trains the model on assistant-generated reasoning and tool calls while masking external tool responses from the prediction loss. Direct preference optimization (DPO) subsequently favors successful trajectories over unsuccessful ones.
The approach targets dependencies across information retrieval, computation, and answer formation rather than treating the final answer as the only learning target.
The main training comparison uses 110 questions from 53 videos excluded from training, with matched inference configurations for adapted models and their corresponding backbones.
| Model capacity | Backbone ACC | TriPhys ACC | Relative improvement |
|---|---|---|---|
| 4B | 46.44% | 54.24% | +16.8% |
| 9B | 46.90% | 54.58% | +16.4% |
| 27B | 49.41% | 58.18% | +17.7% |
Source: Section 6.2, Table 5. ACC values are partial-credit scores multiplied by 100.
For the 27B configuration, the improvement is 8.77 percentage points, equivalent to a 17.7% relative gain. Solution correctness increases from 67.84% to 71.56%.
The cumulative training-stage comparison adds an informative qualification. For the 27B model, SFT improves solution quality without increasing answer correctness; the subsequent DPO stage raises answer correctness. A bicycle-wheel case study similarly illustrates how retrieving an appropriate specification can correct an answer while subsequent optimization improves its solution score.
The case study explains a possible mechanism, but does not establish its frequency. Nor does the cumulative comparison isolate DPO’s independent effect: there is no DPO-only condition, and other trajectory-construction choices are not separately tested.
TriPhys-27B exceeds GPT-5.5 on held-out answer correctness, 58.18% versus 51.14%, but remains below Gemini-3.6-Flash at 59.10%. Gemini also retains a small advantage in overall score, 62.50% versus 62.19%.
What operators should validate before deployment
For a team developing a video inspection assistant, Cognaptus infers a practical evaluation pattern from these results: record and assess the full chain from observation to conclusion.
A task-specific evaluation could require the agent to provide identifiable video observations, retrieved specifications with applicability conditions, selected physical relations, numerical transformations, and a final answer consistent with its units and assumptions. Separate checks would expose whether a failure originated in measurement, external knowledge, computation, or reporting.
This is an engineering recommendation derived from the benchmark’s decomposition, not a demonstrated improvement in industrial inspection performance.
Several boundaries matter before using the results for model selection or deployment approval.
The held-out partition contains only 110 questions, including four physical-judgment questions. Training results cover three Qwen configurations and single runs, without repeated-seed uncertainty intervals or significance tests. Reference solutions can rely on typical physical properties, simplifying assumptions, and measurement tolerances. Language-model-assisted reference review and scoring do not guarantee independent physical correctness.
Video-level separation reduces direct training-test recording overlap, but does not establish reliability across new environments, hardware, or interaction conditions.
Most importantly, the experiments assess video-based question answering with retrieval and computation tools. They do not measure physical robot control, manipulation, or execution success.
An operator evaluating a robot would therefore still need task-specific execution tests before interpreting higher reasoning scores as evidence of safer or more reliable physical behavior.
Better physical reasoning starts with accountable inputs
TriPhysBench makes a specific evaluation gap visible: access to perception, search, and calculation tools does not guarantee that an agent coordinates them correctly.
TriPhys demonstrates that learning complete tool-integrated trajectories can improve measured performance across three model capacities on held-out-video questions. Its solution diagnostics also show why operators should examine the information supporting an answer rather than relying exclusively on the final number.
The next operational question is whether those improvements survive realistic measurement uncertainty, changing specifications, unfamiliar conditions, and eventual physical execution. That remains a separate empirical test.
For now, the strongest actionable contribution is a more disciplined way to evaluate the chain of physical reasoning before trusting its conclusion.
Cognaptus: Automate the Present, Incubate the Future.
-
Zhongxiang Sun and Chenghao Yang and Haojie Ding and Boheng Zhang and Fan Yang and Qingsong Wen and Kun Wang (2026). TriPhysBench: Evaluating Physical Intelligence for Embodied Agents. arXiv:2611.01934. https://arxiv.org/abs/2611.01934 ↩︎