The Lab's Next Move Is the Hard Part

TL;DR for operators

An AI assistant can recognize laboratory equipment, explain chemical principles, and recommend a familiar procedure while still making the wrong decision for the sample in front of it.

LabOPBench evaluates this problem using 756 expert-authored laboratory decision questions. Eight multimodal models achieved normalized rubric scores between 28.8% and 45.8%, with postprocessing averaging just 34.9% across models. These percentages measure accumulated rubric points, not the proportion of fully correct answers.

The recurring weaknesses concern both reading the experimental evidence and selecting actions consistent with it. On a controlled subset, mismatching laboratory images reduced the cross-model mean score from 37.5% to 19.3%.

For R&D managers and laboratory AI developers, the implication is to evaluate assistants on specific experimental decisions rather than general scientific fluency. A credible acceptance test should check whether the model identifies the relevant observations, explains the experimental state, and recommends an appropriate next operation. High-consequence recommendations still require expert verification, and static benchmark performance does not establish readiness for autonomous laboratory control.

A reasonable procedure can still be the wrong procedure

Consider a laboratory team investigating a purification problem. An AI assistant examines the available images and experimental records, explains what might have happened, and recommends a recognizable corrective procedure.

The explanation sounds scientifically reasonable. But the decisive question is whether the recommendation follows from the actual sample condition, previous experimental operations, and available measurements. A procedure that works in one situation may be inappropriate in another.

This is the problem examined by LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making.¹ Rather than testing scientific knowledge alone, the researchers evaluate whether models can use laboratory observations and context to reach appropriate experimental judgments.

The distinction has immediate consequences for organizations evaluating scientific assistants. Model fluency may support an initial demonstration, but it cannot establish that the assistant will choose the correct next action when the experimental evidence is ambiguous, incomplete, or inconsistent with a familiar explanation.

Testing the connection between observation and action

LabOPBench contains 756 open-ended questions built from authentic laboratory materials: 664 organic-synthesis questions and 92 materials-science questions. The evidence includes photographs, instrument interfaces, spectra, chromatograms, microscopy images, and experimental records.

Each question combines an experimental context, relevant observations, and a decision target. Twenty-two contributors with laboratory experience developed the questions and scoring criteria, with two additional independent expert reviews and adjudication when needed.

The central methodological contribution is the question-specific scoring rubric. A response is not rewarded simply for mentioning scientifically valid concepts. Its assessment depends on whether it reaches the correct judgment, uses the supplied evidence, explains the relevant cause, and recommends a suitable operation.

The rubric assigns scores from zero to three, with half-points permitted. A complete, evidence-consistent response can receive three points; a response that reaches the right judgment but omits necessary justification may receive two. Partial observations without a supported conclusion receive less, while incorrect or evidence-contradicting decisions receive zero.

This makes the benchmark particularly relevant to decision-support systems. An assistant that identifies a laboratory observation but recommends an incompatible action has not completed the task successfully.

Eight multimodal models answered the same questions without external tools, conversational history, reference answers, or access to the scoring rubrics. Two independent automated judges, GPT-5.5 and Gemini 3.1 Pro, evaluated the responses against the expert criteria.

Postprocessing exposes the largest performance gap

On the complete benchmark, Claude Opus 4.7 achieved the highest normalized score at 45.8%, followed by GPT-5.5 at 43.6% and Gemini 3.1 Pro at 41.9%. The lowest-scoring model achieved 28.8%.

A normalized score of 45.8% means the model accumulated 45.8% of the available rubric points. It does not mean that 45.8% of its decisions were completely correct.

Performance also varied substantially across laboratory activities.

Source: Section 4.1, Table 2. Values are normalized rubric scores.

Postprocessing was the weakest category for seven of the eight models. It covers operations such as extraction, evaporation, chromatography, and other procedures where the present material state and preceding steps influence the appropriate decision.

It also represents 425 of the benchmark’s 756 questions, including 201 thin-layer chromatography (TLC) questions. Consequently, the overall rankings reflect this substantial postprocessing component rather than equal weighting across laboratory categories.

For an R&D manager considering an assistant for purification troubleshooting, this category-specific weakness is more directly relevant than the model’s overall ranking. An assistant may perform comparatively well in monitoring or materials characterization without being reliable for the particular decisions involved in separating, processing, or analyzing reaction products.

That is a task-selection implication drawn by Cognaptus, not a deployment outcome measured by the benchmark.

The failure often occurs between reading evidence and choosing an action

The researchers conducted expert error analysis on non-full-score responses from the three leading models. Of 266 responses examined on the controlled subset, 159 received at least one error label, producing 257 assignments.

Two categories dominated the model-balanced distribution of assigned labels:

• Evidence Misreading — 40.0%: Relevant experimental observations were misinterpreted or overlooked.
• Evidence-Decision Conflict — 35.8%: The diagnosis or recommended operation was inconsistent with the evidence or the expert rubric.

Together, these represented 75.8% of model-balanced label assignments. These are not error rates across all model responses. The taxonomy is non-exhaustive, permits selected overlapping labels, and describes observable mistakes rather than proving their internal computational causes.

The paper provides two revealing examples.

In one TLC case, the expert judgment identified inappropriate developing-solvent polarity. The model instead treated standard spotting as problematic and attributed the observed pattern to excessive sample loading. The response selected a plausible laboratory cause, but not the cause supported by the observations.

In another case, appropriate LC-MS sampling required collecting a representative solid-liquid aliquot while stirring and then dissolving it in methanol. The model recommended centrifugation or filtration followed by analysis of the clear supernatant. That operation would exclude the solid fraction needed for representative measurement.

These examples illustrate different failure points. In the first, the interpretation of the evidence was wrong. In the second, the proposed operation conflicted with what the measurement needed to represent.

For laboratory decision-support software, this suggests separating checks for observation interpretation, diagnosis, and recommended action. A chemically familiar response should not pass evaluation if its proposed operation contradicts the experimental state.

Changing the images changes the answers, but does not prove visual understanding

A separate image-mismatch ablation tested whether the models’ judgments depended on the supplied visual evidence.

The researchers used the same 100 organic-synthesis questions under two conditions. In the original condition, each question retained its matching images. In the modified condition, those images were replaced with images drawn from other questions in the same fine-grained subcategory.

Question text, experimental context, image count, and scoring criteria remained fixed.

The cross-model mean normalized score fell from 37.5% with matched images to 19.3% with mismatched images, a difference of 18.2 percentage points between the rounded condition means.

Every evaluated model lost performance. GPT-5.5 declined by 24.0 percentage points, Gemini 3.1 Pro by 21.8 points, and Claude Opus 4.7 by 21.0 points.

This ablation establishes that image-question correspondence materially affects the models’ benchmark performance. It does not demonstrate that the models consistently interpreted the correct images accurately. Deliberately mismatched images can also introduce contradictory or misleading evidence.

For laboratory automation developers, the engineering implication is concrete: images, instrument readings, sample identifiers, and procedural records should be checked for correspondence before an AI recommendation is acted upon.

The paper does not test such an alignment-checking system or quantify its potential improvement. It identifies a vulnerability that makes the proposed safeguard worth evaluating.

Human comparisons provide a reference, not a deployment threshold

The researchers also compared model performance with five undergraduate and five PhD participants on the same controlled 100-question organic-synthesis subset.

The undergraduate group averaged 31.0%, while the PhD group averaged 45.9%. Claude Opus 4.7 scored 43.7% and GPT-5.5 scored 43.5%. Seven of the eight models exceeded the undergraduate mean, but every model remained below the PhD mean.

The results suggest that leading models can perform competitively with some human participants on isolated, evidence-grounded laboratory questions. However, five participants per group cannot establish broad equivalence with laboratory expertise.

The benchmark also contains no physical interventions or subsequent observations. It therefore cannot determine whether a model would recognize the consequences of its recommendation, correct an unsuccessful operation, or recover from a changing experimental state.

The scoring procedure has its own validation evidence. On a stratified sample of Claude Opus 4.7 responses, the combined automated judges achieved a mean absolute error of 0.165 points on the three-point rubric relative to the expert mean, with quadratic-weighted kappa of 0.9161.

That supports scoring consistency for the validated sample, but not universal judge reliability across every model. GPT-5.5 and Gemini 3.1 Pro were themselves evaluated models as well as judges, and possible model-specific judging effects were not comprehensively excluded.

Building acceptance tests around laboratory decisions

For laboratory AI product teams, Cognaptus draws a practical implementation approach from LabOPBench: construct acceptance tests around the actual decisions the assistant is expected to support.

This framework is a Cognaptus inference inspired by the benchmark’s construction and observed errors. Its effectiveness, implementation cost, and impact on real experiments remain unmeasured by the paper.

A sensible evaluation process would begin with representative cases from the laboratory’s intended workflow, scored against expert-defined judgments. Postprocessing scenarios deserve particular attention when purification or sample preparation is part of the proposed product.

Only after static recommendations have been tested should teams separately assess sequential decisions, actual execution, post-action measurements, and recovery from unsuccessful interventions.

Laboratory assistance remains a narrower claim than autonomy

LabOPBench makes a substantive contribution by evaluating a capability that broad scientific benchmarks do not necessarily capture: connecting actual laboratory observations to the correct experimental decision.

Its results indicate that even strong multimodal models can struggle with that connection. The failures are not confined to missing scientific terminology or inability to identify an instrument. They include misreading relevant evidence and recommending operations that do not follow from it.

For organizations deploying laboratory AI, the appropriate next step is therefore task-specific validation rather than treating general model competence as sufficient. Expert-authored decision criteria, evidence-alignment checks, and review of high-consequence recommendations provide a reasonable starting design.

The boundary remains clear. LabOPBench evaluates static, offline judgments in organic synthesis and materials science. It does not measure robotic execution, iterative replanning, physical safety under deployment, or reliable closed-loop laboratory autonomy.

An AI assistant may become valuable in laboratory operations before it can independently run an experiment. Establishing that value requires demonstrating which decisions it supports correctly, under which evidence conditions, and with what human oversight.

Cognaptus: Automate the Present, Incubate the Future.