The Molecule Was Right. The Reasoning Was Not.

TL;DR for operators

Chemistry teams should stop treating a correct molecule, reaction product, or ranked option as proof that an AI system reasoned chemically. That is the comfortable interpretation. It is also, inconveniently, the one ChemCoTBench-V2 was built to dismantle.

The paper introduces a benchmark that evaluates chemical language models at three separate levels: final-answer correctness, template adherence, and step-wise chemical validity. The important move is not “add more benchmark rows.” The move is to force the model to expose intermediate chemical commitments—rings, scaffolds, fragments, reaction types, edit plans, condition rankings, product constructions—and then check those commitments with deterministic chemistry rules or verified reference traces.¹

The result is a sharper diagnosis of current chemical LLMs. Models often produce beautifully formatted reasoning traces. They often satisfy the requested template. They sometimes even produce the correct final answer. Then Layer 3 quietly points out that a scaffold is wrong, a ring count does not survive the edit, a condition ranking is internally neat but chemically mismatched, or a product was reached through a benchmark-inconsistent reaction state. Science, meet paperwork. Paperwork wins.

For business use, the practical lesson is clear: do not merely benchmark the answer. Benchmark the state transitions. In pharma, materials, and chemical R&D workflows, answer-level evaluation can tell you whether a model got lucky. State-level verification tells you where it stopped being trustworthy. That difference matters for model selection, tool routing, audit logs, expert review, and deciding whether a system should be allowed anywhere near an expensive wet-lab queue.

The boundary is equally important. ChemCoTBench-V2 is strongest where intermediate chemical states are expressible in 2D molecular or reaction representations and can be checked by rules, oracles, and verified references. It is not a proof that LLMs can run laboratory planning, reason over 3D conformations, perform quantum chemistry, or discover drugs without adult supervision. A mildly tragic clarification, but apparently still necessary.

A correct answer is not the same as a valid chemical trace

A familiar enterprise pattern is now arriving in scientific AI: a model produces a plausible output, the output passes a superficial check, and everyone pretends the process underneath must have been competent. In customer support, this gives you a confident wrong refund policy. In finance, it gives you a clean but unsupported spreadsheet explanation. In chemistry, it can give you a correct-looking SMILES string supported by a chemically broken trace.

That is the misconception this paper attacks. Chemical reasoning is not just selecting a final molecule. It is maintaining a set of commitments about a molecular graph or reaction context while moving through a task. If the model says it identified a scaffold, removed one group, added another, preserved ring topology, predicted a product, and then gave an answer, those claims can be inspected. They are not poetic interior monologue. They are state claims.

ChemCoTBench-V2 reframes evaluation around those claims. Rather than asking only “was the answer right?”, it asks three questions:

This separation is the paper’s mechanism. It turns “reasoning” from a charming paragraph into a set of inspectable chemical objects. That is less glamorous than a frontier-model leaderboard, which is precisely why it is useful.

The benchmark works by making the model put its chemistry on the table

ChemCoTBench-V2 contains 5,620 active evaluation samples across 18 reporting tasks, built from 31 fine-grained chemical tasks. The four task families are molecular understanding, molecule editing, molecular optimization, and reaction prediction. This matters because the benchmark does not merely test static chemical trivia. It includes tasks where a model has to update or preserve molecular and reaction states across a structured trace.

The paper’s construction pipeline is deliberately strict. Molecules and reactions are drawn from public chemistry resources and task-specific pools, then filtered with RDKit-based sanitization and canonicalization. Molecule-editing examples are derived from real reactant–product changes and rewritten into site-specific edit instructions. Condition-ranking labels are shuffled so that a model cannot exploit a fixed label order. The active benchmark is selected from a larger 12,600-sample construction pool, leaving a balanced evaluation set that is computationally feasible but still broad enough to expose task-level differences.

The real trick is the formal template. Instead of allowing the model to write generic chain-of-thought prose, the benchmark asks it to fill task-specific structured fields. A substitution edit may require anchor identification, removed group, incoming fragment, product construction, heavy-atom verification, ring verification, and final answer. A condition-ranking task may require reaction class, decision factor, pairwise comparisons, pairwise preferences, global ranking, top-two support, and answer. A molecular-optimization task may require scaffold extraction, edit-plan validity, product validity, scaffold preservation, and functional-group change consistency.

That gives the verifier something to grip. A free-form sentence like “the scaffold is preserved” is cheap. A parsed field saying which scaffold was identified, which molecule was predicted, and whether preservation holds under an RDKit comparison is more expensive. Conveniently, it is also harder to bluff.

The paper’s evaluation setup runs eight frontier models through the same prompts, parsers, and verifiers. The model suite includes reasoning-oriented and standard instruction-following systems, with all API evaluations performed in May 2026. Each model receives 5,620 prompts, for roughly 44,960 model calls excluding retries. The authors do not collapse all task families into one grand score, because Layer 1 metrics differ across tasks. Good. A single magic score for chemical reasoning would have been tidy, marketable, and mostly useless.

The first failure mode is protocol compliance without chemical validity

The most operationally useful result is that models are much better at following the requested format than at maintaining valid chemical states.

Layer 2 scores are often near-perfect. In molecular understanding, the paper reports Layer-2 State Score min/median/max of 0.9367/0.9992/1.0000. For molecule editing, the range is 0.8900/0.9836/1.0000. For molecular optimization, it is 0.9520/0.9922/1.0000. For reaction prediction, even with a lower minimum, the median is still 0.9967.

Then Layer 3 walks in and ruins the mood.

Across task families, the paper reports average Layer-3 Type-I/Type-II scores of only 0.310/0.319 for molecular understanding and 0.386/0.226 for reaction prediction. Molecule editing shows the same gap: Layer 2 averages 0.970, but Layer 3 drops to 0.648/0.543. The model can fill the form. The form can still be chemically wrong.

One result captures the point with painful efficiency: SMILES equivalence reaches 86.9% Layer-1 accuracy, but only 29.9% Layer-3 Type-II all-match. In plain language, models can often decide whether two molecular strings refer to the same structure, while failing to support that answer with intermediate commitments that match the verified benchmark state. The final answer is not a receipt. It is a claim. The paper asks for the audit trail.

For business users, this is not a minor academic distinction. If an AI chemistry assistant is used for triage, ideation, or pre-screening, the final output may be less important than the reliability of the route that produced it. A model that gets some answers right for unstable reasons is not necessarily useless. But it should be routed differently. It may be suitable for suggestion generation, but not for autonomous decisioning. It may help populate candidates, but it should not silently graduate candidates into costly follow-up workflows.

The second failure mode is local chemistry without persistent state tracking

The paper’s evidence is not merely “LLMs fail chemistry.” That would be too broad to help anyone. The more interesting finding is that models often possess local chemical heuristics but struggle to preserve and update structured commitments over time.

In molecular understanding, the paper distinguishes local pattern recognition from graph-topology reconstruction. Functional groups and some equivalence judgments can be more tractable because they often rely on local or string-level cues. Ring counting and scaffold extraction are less forgiving because they require the model to maintain an explicit graph object. The checkpoint logs make this concrete: ring-count failures concentrate in ring-pattern identification at 67.0% and total-count validation at 50.4%, while Murcko scaffold failures spike in substructure containment at 73.0%.

That is not a generic “model bad” result. It is an error map. If your chemical AI system fails at scaffold containment, you do not need another inspirational prompt about being a careful chemist. You need a scaffold verifier, tool call, fallback path, or a model architecture that can keep graph-state commitments intact.

Molecule editing shows the same pattern in a more operationally familiar form. The benchmark’s editing tasks are based on reaction-derived local graph changes: add, delete, and substitute. Layer-1 exact match can be high, especially for constrained add/delete operations. But the step logs show failures in ring-count consistency, heavy-atom accounting, product construction, and final-answer fields. Add/delete errors concentrate in ring-count consistency at 29.8%/27.6% and heavy-atom accounting at 17.1%/16.5%. Substitution produces Type-II mismatches at product construction and final-answer fields at 32.4%/34.2%.

The model can often describe the local edit. It can lose track of what that edit does to the molecule as a whole. That is the difference between “looks like chemistry” and “maintains a chemical state.” One is useful for demos. The other is useful for work.

Optimization exposes the cost of coupled constraints

Molecular optimization is where the benchmark becomes especially relevant for R&D leaders, because this is the use case everyone wants to accelerate: propose better molecules under constraints. The paper’s result is not that models are hopeless. It is worse, or better, depending on one’s tolerance for nuance: they are selectively competent.

Single-objective physicochemical optimization is comparatively strong, with an average success rate of 83.5%. Biological target optimization is harder, at 45.2% average success. But the sharp collapse appears in dual-objective settings. The authors report that the marginal success rate for each objective remains about 71%, while joint success drops to 9.8% for dual physicochemical optimization and 6.1% for dual biological-target optimization.

That pattern matters. It suggests the model is not merely ignorant of all chemical edits. It can find locally useful transformations. The problem is composition: satisfying multiple commitments simultaneously while keeping the molecule coherent. This is where enterprise AI systems often break as well. They can satisfy one constraint, then another, then quietly violate the first while congratulating themselves on the second. Very human, frankly. Also very expensive.

Layer 3 clarifies the failure. Across molecular-optimization groups, functional-group change verification is almost always satisfied, at about 99% oracle-verified consistency. The weakest step is scaffold-preservation verification, with failure rates from 73.0% to 83.2%. That means models can often express a local functional-group change but fail to preserve the global scaffold state they claim to be preserving.

For a business workflow, this suggests a very practical routing rule. Do not ask whether the model generated a molecule that improved a target property. Ask whether it improved the property while preserving the structural commitments that make the candidate relevant. If the scaffold claim fails, the downstream decision should change. Otherwise the system is not optimizing molecules. It is playing molecular whack-a-mole with a lab budget.

Reaction prediction separates valid syntax from grounded reaction context

Reaction prediction adds another version of the same story. Models can produce outputs that are syntactically valid, internally consistent, or chemically plausible in isolation, while still failing to bind the output to the provided reaction context.

Condition ranking is the cleanest aggregate example. The paper reports 99.4% Type-I validity for condition-ranking traces, but only 11.6% Type-II benchmark-state agreement. This means models often produce rankings that are structurally well-formed and internally coherent: all labels appear, pairwise preferences make sense, the ranking is a valid permutation, the top-two support is aligned with the stated factor. Yet the ranking does not match the verified reference state.

The appendix case study makes the point nicely. A model ranks three condition sets for a deoxyfluorination reaction. It chooses “base” as the decision factor, compares all pairs, constructs acyclic pairwise preferences, and outputs a ranking consistent with its own trace. Layer 2 is perfect. Type-I passes. But the experimentally induced reference ranking is the reverse order, so Type-II fails. The paperwork is impeccable. The chemistry is not.

This is a useful distinction for deployment. A local symbolic verifier can catch malformed outputs, impossible fields, invalid molecules, and inconsistent rankings. It cannot, by itself, prove that the model picked the right reaction abstraction or condition preference. That requires benchmark-state agreement, empirical data, expert review, or a task-specific predictive model. The correct architecture is not “LLM plus optimism.” It is LLM plus verifiers plus domain data plus routing. Less charming on a slide. More likely to survive contact with chemistry.

The ablation shows scaffolds help, but they do not solve the state problem

The paper includes a prompt ablation on DeepSeek-V3.2 comparing Direct, Template, and Template+Anchor. This is an ablation, not a second thesis. Its purpose is to test whether structured prompts and safe intermediate anchors help models maintain chemical reasoning paths.

They do help. In molecular optimization, the average success rate rises from 15.9 under Direct prompting to 31.5 with Template and 32.7 with Template+Anchor. In molecular understanding, some tasks improve dramatically: ring-system scaffold accuracy moves from 0.253 to 0.420 to 1.000, and SMILES equivalence accuracy from 0.637 to 0.863 to 1.000.

But the ablation is not a victory parade. Ring-count MAE worsens slightly from 1.340 under Direct to 1.457 with Template and 1.450 with Template+Anchor. Dual-objective optimization remains difficult. The improvement from Template to Template+Anchor is also modest for optimization overall. The fair interpretation is that explicit scaffolds and anchors can reduce some planning burden and improve state exposure, but they do not eliminate the deeper bottleneck: maintaining correct chemical commitments across a long trace.

For operators, this is the difference between prompt engineering and system design. Templates are useful. Anchors are useful. They are not a substitute for deterministic checks, chemical tools, oracle calls, and escalation pathways. Prompting can make the model write down its commitments. Verification decides whether those commitments deserve to live.

The appendix matters because it tells you what kind of evidence this is

The paper’s appendices are not decorative storage. They clarify what should and should not be inferred from the main results.

The expert-validation result is particularly important. A deterministic verifier that agrees with adjudicated expert labels 87.4% of the time is useful, but not divine. The disagreements cluster in tasks with path multiplicity: molecular optimization, condition ranking, and retrosynthesis. That is exactly where one should be careful about treating Type-II benchmark-state agreement as the only valid chemical rationale. The paper itself is explicit about this: Type-II is benchmark-state agreement for closed-answer tasks, not exhaustive validation of all possible chemical rationales.

That nuance is not a weakness. It is what makes the benchmark serious. The authors are not claiming to evaluate all chemistry. They are claiming to evaluate verifier-addressable intermediate commitments under defined task constraints. That is a narrower claim. It is also the kind enterprises can actually operationalize.

What Cognaptus infers for chemical AI deployment

The direct paper result is an evaluation result: current frontier models often separate final-answer success, formatting compliance, and step-wise chemical validity. The business inference is that chemical AI systems should be evaluated, monitored, and routed according to the same separation.

A practical enterprise architecture should look less like this:

Prompt model -> accept final answer -> hope the explanation was meaningful

and more like this:

Prompt model
  -> require formal intermediate commitments
  -> parse states
  -> run deterministic chemistry checks
  -> compare closed-answer states where reference traces exist
  -> call oracles/tools for open-ended optimization constraints
  -> localize first failure
  -> route to accept, revise, tool-assisted retry, or expert review

That architecture changes the economics of trust. It does not make the model magically reliable. It makes unreliability cheaper to locate.

For pharma and materials workflows, this can matter in at least four ways.

First, model evaluation becomes task-specific. A model that performs well on SMILES equivalence may still be weak on scaffold extraction or dual-objective optimization. Buying or deploying “the best chemistry model” is therefore the wrong procurement question. The right question is: which model maintains the states your workflow actually depends on?

Second, failure localization becomes a product feature. If a molecule-edit trace fails at ring accounting, the system can retry with a graph tool. If a condition-ranking trace passes Type-I but fails benchmark-state agreement, the system can route to a reaction-condition model or chemist. If scaffold preservation fails during optimization, the candidate can be rejected before consuming expert time. The point is not merely higher accuracy. The point is controlled handoff.

Third, auditability becomes concrete. A logged final answer is weak evidence. A logged trace with verifier outcomes—scaffold extracted, product parsed, ring delta failed, Type-II reaction class mismatch—creates an audit artifact. This is useful for internal QA, model comparison, regulated workflow design, and post-hoc analysis. It also reduces the temptation to use a polished explanation as a compliance object. Please do not do that. The explanation has not earned it.

Fourth, prompt engineering becomes subordinate to verification. The ablation shows that structured templates help. But the larger message is that templates are only valuable because they expose checkable states. A template without a verifier is just a better-looking hallucination container.

Where the result applies, and where it does not

The boundary of this paper is not a footnote; it is part of the operating model.

ChemCoTBench-V2 is strongest for tasks where intermediate states can be represented as structured 2D molecular or reaction commitments. That includes SMILES validity, SMARTS matching, ring counts, heavy-atom arithmetic, scaffold containment, charge balance, atom conservation, scaffold preservation, product construction, condition ranking, and reference-aligned closed-answer states.

It is weaker wherever chemistry becomes open-ended, path-multiple, physically grounded, or experimentally contingent. Molecular optimization already shows this tension. The benchmark handles open-ended optimization with oracle-verifiable state constraints rather than strict trace matching, which is sensible. But that also means the benchmark should not be interpreted as proving a model can design viable drug candidates under real medicinal chemistry constraints.

The paper does not establish competence in 3D conformational reasoning. It does not establish laboratory feasibility. It does not validate synthesis planning as a deployed system. It does not cover quantum chemistry. It does not replace expert review. The authors state that the released benchmark is intended for non-commercial research evaluation of LLM chemical reasoning traces, not as a synthesis planner, drug-design system, laboratory recommendation tool, safety-decision system, or substitute for expert chemical review.

Those limitations do not reduce the paper’s usefulness. They define it. The value is not that ChemCoTBench-V2 solves chemical AI. The value is that it gives teams a disciplined way to inspect one of the places where chemical AI currently pretends to be better than it is.

The operational lesson: evaluate the molecule, then interrogate the path

The memorable result from this paper is simple: the answer can be right while the chemical state trace is wrong. That should change how scientific AI systems are benchmarked.

Outcome-only evaluation treats the model as an answer machine. ChemCoTBench-V2 treats it as a state-transition system. That is the better abstraction for chemistry, because chemical work depends on preserving structure, constraints, reaction context, and molecular identity across steps. A model that loses those commitments is not “reasoning with chemistry.” It is producing chemical-looking language while the graph quietly wanders off.

The business implication is not to abandon LLMs in chemistry. That would be melodramatic, and melodrama has a poor validation curve. The implication is to stop trusting final answers as evidence of process competence. Require intermediate commitments. Verify them. Localize failures. Route accordingly.

In other words: the model may remember the answer. The verifier remembers the molecule.

Cognaptus: Automate the Present, Incubate the Future.