Stocked but Not Synthesizable: URSA Tests the Chemistry Inside the Route

TL;DR for operators

A team choosing a synthesis-planning system may reasonably treat a route that ends in purchasable starting materials as nearly ready for chemical review. URSA shows why that assumption is risky: reaching stocked compounds proves that a route graph can terminate in available inputs, not that the proposed reactions are chemically plausible.

On URSA’s difficult target set, DreamRetroer completed that graph-and-stock test for 95% of targets, while Kimi K2.5 reached 97%. Once every transformation also had to show precedent-supported reaction logic and a compatible functional-group context, their success rates fell to 9% and 5%, respectively. URSA calls these stages Solv-0 and Solv-2.

The operational consequence is straightforward. Do not choose a system, send routes to chemists, or budget verification work from completion rates alone. Require chemistry-aware screening—such as URSA’s precedent-based ChemCensor check—before benchmark claims influence experimental planning. Even a route that passes this screening still requires chemist approval and does not establish yield, safety, purification feasibility, or laboratory success.

A completed route can still fail as chemistry

Consider a pharmaceutical R&D team comparing synthesis-planning systems. One supplier reports that its model produces complete routes for 95% of difficult targets. Another reports 65%. Without a common definition of completion, the first system appears decisively stronger.

The trouble begins when “complete” means only that the route graph ends in compounds listed as commercially available. That test can confirm connectivity and stock termination while ignoring whether the transformations between nodes are chemically plausible. A planner may navigate from target to catalog without preserving viable reaction logic along the way.

DreamRetroer reached 95% on the graph-and-stock criterion but only 9% when each reaction also had to satisfy reaction-center and functional-group plausibility checks. Kimi K2.5 fell from 97% to 5%. The correction changes the procurement decision. A high stock-termination rate answers whether the system can reach available inputs, not whether a medicinal chemist should review or execute the route.

URSA turns route quality into staged gates

URSA builds a common evaluation pipeline for heterogeneous planners rather than asking every system to reproduce one historical synthesis. That distinction matters. Exact route matching can reject a novel route even when its chemistry is defensible; stock termination can accept a route whose chemistry is not.

The framework standardizes conventional planner outputs and LLM-generated routes into a shared representation, then applies several gates:

URSA also generates collapsed route variants before scoring. This measurement safeguard reduces the risk that different representations of the same synthetic operation are mistaken for bad chemistry.

Every reaction is then scored with ChemCensor, a deterministic validator that searches for precedent around the reaction center and evaluates functional-group context. Route-level metrics aggregate those step judgments. The benchmark selects one best route per target by prioritizing Solv-2 satisfaction, then average ChemCensor score, and then shorter length.

The headline tables therefore describe the best retained route per target, not candidate-quality distributions, ranking reliability, or route diversity.

The evaluator is part of the benchmark

A chemistry-aware benchmark is only as credible as the judge applying its chemical criteria. URSA therefore tests ChemCensor rather than treating the evaluator as neutral infrastructure.

The comparison uses 1,000 expert-labeled reactions, balanced between 500 plausible and 500 implausible cases. ChemCensor achieved 0.96 accuracy and 0.92 MCC. Gemini 3.1 Pro, the strongest LLM judge in the reported comparison, reached 0.83 accuracy and 0.67 MCC.

The LLM judges recalled plausible reactions at 0.90 to 0.96, but implausible reactions at only 0.50 to 0.73. Their approval bias can send defective steps into human review and make a planner appear safer than it is.

ChemCensor is not chemical ground truth; its decisions depend on precedent coverage and scoring rules. Still, the expert-label experiment supports it more strongly than direct LLM judgment for this benchmark.

For model-governance teams, evaluator choice should therefore appear in the benchmark specification alongside target set, stock catalog, prompt, model version, and route cap. Changing the judge can change the apparent capability being purchased.

Specialized planners lead when the full route is scored

The route-level comparison covers proprietary foundation models, open-weight models, and conventional computer-assisted synthesis planning systems on two 100-target sets. One set contains difficult, prospective-style medicinal-chemistry targets; the other contains approved drugs and clinical candidates. LLMs could submit up to ten routes per target.

On the difficult set, no system passed Solv-2 for more than one-third of targets. RetroChimera-MCTS led at 32%, followed by AZF-MCTS at 29%, Retro*-0-LR at 22%, and GPT 5.5 at 21%. On the drugs-and-clinicals set, RetroChimera-MCTS reached 60%, while AZF-MCTS and AZF-Retro* reached 54%.

The paper directly supports three conclusions.

First, specialized retrosynthesis systems remain highly competitive. AZF-MCTS outperformed every evaluated LLM on both main target sets, and RetroChimera-MCTS produced the highest Solv-2 result on each.

Second, local reaction judgment and global route construction are separate capabilities. A model may classify individual reactions reasonably well yet fail to maintain mutually compatible chemistry across a multistep route. Each additional step creates another opportunity for unsupported transformation logic, incompatible functional groups, or accumulated incoherence.

Third, model prestige is a poor substitute for task-specific validation. Some foundation models produced respectable results, particularly GPT 5.5, but the overall comparison does not support a default assumption that a broader frontier model will dominate a narrower planner.

The observed cost results reinforce the procurement point, although they are less standardized. AZF-MCTS beat all evaluated LLMs at relatively low cost, and RetroChimera-MCTS remained less expensive in the reported runs than proprietary frontier models. Deployment mode, API pricing, and infrastructure visibility limit these comparisons to screening rather than durable cost forecasts.

Precedent coverage changes scores more than rankings

The main benchmark uses the open U2 database, derived from roughly 1.4 million USPTO reactions. A supplementary configuration adds about 3.5 million Pistachio precedents.

This comparison is best read as a robustness test of the measurement system, not as a second leaderboard thesis. Expanding the database raised absolute Solv-2 values for many planners because more transformations found relevant support. The broad ranking changed little.

Precedent coverage therefore affects measured plausibility, while the broad comparison was not merely an artifact of the smaller database.

For benchmark owners, corpus investment is part of evaluation quality. Broader coverage can reduce false rejection of valid but underrepresented chemistry. It can also import proprietary-data dependencies, historical publication bias, and uneven representation across reaction families. A stronger validator database improves screening only within the chemistry it adequately covers.

How pharmaceutical teams can use the hierarchy

Cognaptus infers a practical governance pattern from the benchmark: convert the Solv-N hierarchy into explicit workflow gates rather than one aggregate planner score.

At procurement, require vendors to report route integrity, Solv-0, Solv-1, and Solv-2 separately on targets that resemble the buyer’s portfolio. A single completion percentage should not qualify a product for medicinal-chemistry use.

At runtime, route low-scoring or unsupported plans directly to rejection or regeneration. Send higher-scoring plans to chemists with the reaction-level evidence attached, including which steps lack precedent and which functional-group contexts triggered concern.

For model routing, use specialized CASP systems as the default engines for complete-route generation when they lead on the relevant target class. Foundation models may still support target decomposition, literature interaction, explanation, or alternative-strategy generation, but those roles should be validated separately from end-to-end route planning.

For governance, record the exact benchmark configuration: model version, prompt, target set, stock definition, maximum routes, validator version, precedent database, and route-selection rule. Without that record, a later score cannot be reproduced or compared fairly.

The likely return is cheaper rejection of weak routes before chemist time and laboratory capacity are committed.

Solv-2 is a screening gate, not a laboratory certificate

URSA evaluates computational chemical plausibility. It does not evaluate reaction conditions, catalysts, solvents, purification, safety, yields, scale-up behavior, or successful execution. Those missing elements define the boundary between Solv-2 and experimental executability.

The findings are also conditional on two 100-target sets, a shared stock catalog, selected model versions and prompts, a ten-route cap for LLMs, ChemCensor v1.2.0, and the U2 or U2P2 precedent configuration. Another target distribution or reaction corpus could alter absolute scores and perhaps rankings.

Finally, choosing one best route per target leaves several operational questions unresolved: whether a system ranks its own candidates well, whether it offers useful route diversity, how failures cluster across chemistry classes, and whether a chemist can efficiently repair near-valid plans.

These boundaries do not weaken the central contribution. They identify the proper authority of the benchmark. URSA can support comparative screening, procurement diligence, and review prioritization. It cannot authorize laboratory execution.

Measure the chemistry before buying the completion rate

URSA exposes a category error in retrosynthesis evaluation: reaching purchasable inputs is treated as evidence that the route itself is chemically sound. The reported drops from Solv-0 to Solv-2 show that the two properties can diverge by more than 80 percentage points.

The stronger procurement question is not which planner finishes the most route graphs. It is which system produces the highest proportion of chemically defensible routes, on the buyer’s targets, under a documented evaluator, at an acceptable review and compute cost.

That standard favors staged evidence over a single leaderboard number. It also keeps the final decision where it belongs: automated systems can filter and prioritize; chemists still approve the experiment.

Cognaptus: Automate the Present, Incubate the Future.