From Pre-Audit to Proof: How Technical AI Earns More Decision Rights

TL;DR for operators

Do not assign technical AI authority from a headline accuracy number. Use AI freely for repeatable first-pass checks where the rule is explicit and the cost of a false flag is low. Keep human experts responsible when the decision depends on system context, feasibility, architecture, necessity, or trade-offs. Where a claim can be rerun, make execution part of the evidence standard: a reproduced calculation deserves more weight than an unsupported model judgment. Even then, preserve review for interpretation, setup choices, and citations.

A technical AI system is easier to trust when its output can be compared with something outside the model. In one 2026 computational-physics study, an LLM agent reproduced 75.8% of 571 deduplicated quantitative claims within 5% of the published value and 83.2% within 10%.¹ Those numbers matter because they measure contact with an external calculation, not fluency.

They do not settle the trust question. A separate requirements-engineering study found that an LLM could agree closely with engineers on a selected quality-assessment task while still emphasizing different reasons for revision.² The physics study, meanwhile, documented an interpretation error and citation-fidelity lapses even after successful computational execution. Agreement, reproduction, and sound judgment are related, but they are not interchangeable.

That distinction affects deployment. If every model output is treated as having the same evidentiary status, organizations will automate some decisions that still require domain context and leave other, more mechanical checks unnecessarily expensive. The useful unit of governance is therefore not “the AI system” as a whole. It is the individual judgment, the evidence available for that judgment, and the party allowed to act on it.

First-pass review is a good place to delegate

The requirements-engineering evidence defines the front end of this allocation.

The study evaluated LLMs against seven INCOSE-aligned qualities such as unambiguous, singular, complete, feasible, and verifiable. In its selected 20-requirement human comparison, Claude Sonnet 3.5 reached 85.0% agreement with individual engineer assessments, with a reported 95% bootstrap confidence interval of 81.2% to 88.3%. GPT-4o and Llama 3 were much lower in that experiment. On a separate 969-item software-requirements dataset, relative strengths changed again: the models showed different functional versus non-functional classification biases and different category-level strengths.

The operational lesson is not that one model “understands requirements” better in general. The paper gives evidence for a narrower use: automated first-pass review can repeatedly flag linguistic ambiguity, compound requirements, missing measurable parameters, and similar structural defects. This is what a pre-audit should mean in practice: a machine review that identifies likely problems before a human owns the final decision.

The boundary appears when formal quality rules stop being enough. The authors report a rationale gap between model and engineers even when both sides agree that a requirement should change. Models tend to focus on formal violations; engineers more often invoke missing context, ambiguity, or conceptual intent. Human assessments also varied substantially across requirements, with the share of engineers identifying an issue ranging from 10% to 84%.

For an engineering manager, that makes expert disagreement part of the system rather than an inconvenience to average away. A model can reduce the volume of routine review without becoming the authority on whether a requirement is necessary, technically feasible, economically realistic, architecturally appropriate, or complete relative to stakeholder intent.

Execution can raise the evidence level

The computational-physics study shows that the delegation boundary can move when technical claims are executable.

Across 111 open-access papers using Quantum ESPRESSO as the primary computational tool, the agent did more than read and classify text. It ran calculations, compared outputs with published results, and recorded where concerns became substantive. Under the authors’ corrected phase-coding instrument, 85 of 88 critique instances were classified as becoming substantive only after execution. A blinded recoding produced a lower 71.6% execution-bound share, so the exact percentage is sensitive to the coding rule and should not be treated as a causal estimate.

That qualification matters, but the mechanism is still operationally relevant. A concern such as “this result looks suspicious” has low evidentiary value on its own. If the system can rerun the calculation and produce a conflicting numerical result, the concern becomes inspectable by another person or machine. The model has less room to rely on a plausible narrative because the output is constrained by a physical workflow.

The depth case makes this more concrete. A verified multi-code reproduction pipeline was handed to fresh Review and Reflect agents. The resulting follow-up work identified contact-resistance and antimony-doping attacks on a 5 nm device claim that were absent from the published peer-review concern set. After further calculations, the final graded conclusion was that 7 nm was robust, 6 nm marginal, and 5 nm failed.

This is execution-grounded scrutiny: checking a technical claim by running the relevant calculation or workflow and comparing the result with the claimed outcome. It gives AI a path from screening toward deeper technical review without asking operators to trust the model’s prose alone.

But execution is not the same as technical authority

The same physics paper also shows why tool use should not be mistaken for a completed assurance process.

Workshop review caught an interpretation error in the agent-produced phonon analysis, and the paper documents citation-fidelity problems. The authors explicitly limit the claim: execution constrains numerical statements, but it does not guarantee correct physical interpretation, good setup choices, or faithful citation.

There is also a harness problem. In a controlled 15-paper ablation, compact command and pseudopotential guidance plus a consult-before-refuse rule eliminated a targeted class of false refusals and broadened attempted phonon workflows. That result suggests that some apparent model incapability is actually workflow incapability: the system may possess enough base competence but fail to use available tools because its operating envelope is poorly specified.

For R&D leaders, this changes what should be tested. Model evaluation alone is insufficient. The assurance target includes tool knowledge, solver reliability, compute orchestration, traceability, and review behavior. The study’s trace audit found no observed figure-open-check-regenerate loop in 61 inspected traces, which is a concrete reminder that producing an artifact and inspecting that artifact are separate system capabilities.

A tiered operating model for technical AI

The two papers together support a governance model based on evidence thresholds and decision-right allocation: specifying which judgments AI may make independently, which require expert review, and which require stronger external evidence before action.

This table is a Cognaptus inference, not a tested intervention from either paper. Neither study measures downstream project cost, defect rates, research productivity, or safety outcomes under this governance design. The combined evidence supports the allocation logic, not a universal threshold for autonomy.

The affected users are engineering managers, scientific-computing teams, QA leaders, and research-governance owners. Their decision is not simply whether to deploy an LLM. It is whether a particular output is allowed to trigger a revision, approve a requirement, challenge a published result, allocate compute, or close a review. The condition for greater autonomy should be correspondingly specific: explicit rules for routine checks, expert context for meaning-heavy judgments, and rerunnable evidence for claims that can be externally tested.

What to measure before expanding autonomy

A deployment team can turn this into a measurable review program.

Track agreement separately from rationale. Measure whether the model and expert reached the same label, but also whether they identified the same defect or relied on compatible evidence. Record false refusals and unattempted workflows, because missing tool use can look like reasoning failure. For executable tasks, preserve inputs, outputs, logs, and comparison artifacts so a disputed claim can be rerun. Finally, audit the stages execution does not validate: setup assumptions, interpretation, visual outputs, and citation fidelity.

The key operational shift is from asking whether the model is “accurate enough” to asking what kind of evidence supports each action the model is permitted to influence. The requirements study supports broad delegation of repeatable structural review while retaining expert control over context. The physics study shows that executable evidence can extend AI participation further into technical scrutiny, but not all the way to unreviewed authority.

Technical AI earns more decision rights when its judgments become more externally checkable. Accuracy is one input to that decision. It is not the policy.

Cognaptus: Automate the Present, Incubate the Future.