Trust Signals Don’t Check Themselves: Put Verification Before the Install

TL;DR for operators

If an AI coding assistant can install software, do not assume it will inspect the evidence already available about where that software came from. In Pengyin Shan’s pre-registered demand-side audit,¹ only 9 of 1,920 registered trials retrieved a relevant trust signal before the install-or-decline decision. Across 2,114 completed registered and supplementary trials, not one assistant executed a signature or attestation verification command.

The distinction matters. The paper’s registered “verification-positive” outcome means that the assistant retrieved a relevant signal before deciding. It does not mean that a signature, build attestation, or other provenance claim was cryptographically checked. Actual verification happened zero times.

For organizations deploying coding assistants, the control should therefore sit in the execution layer: retrieve the required evidence, verify it against an expected identity or policy, block installation when the check fails, and log the result. The experiment does not establish that one model, vendor, or trust-signal format is superior. It establishes something more basic in the tested setting: publishing provenance information does little for an automated installation decision when the workflow never consumes it.

The repository can contain the evidence and the assistant can walk past it

Consider a routine installation request. The repository already contains machine-readable evidence about the software: perhaps an inventory of components, a signed release, information about how the build was produced, or a declaration of the official distribution channel. A capable coding assistant has access to the files and is allowed to inspect them before running installation commands.

The security mechanism only becomes operational if the assistant crosses a sequence of steps: find the evidence, inspect it, verify the relevant claim, and use the result before installation.

Shan’s study tests the beginning of that sequence directly. Six open-source HPC and quantum-computing projects were placed into nine controlled repository conditions. These included a clean control, an SBOM, valid or wrong-issuer signed releases, valid or wrong-issuer attestations, an official-channel declaration, all four valid signals together, and deliberately inconsistent metadata.

An SBOM is a machine-readable inventory of software components. An attestation is a structured statement about how software was produced that can be checked against an expected identity or process. Together with signatures and channel declarations, these are provenance signals: evidence intended to help a consumer establish what the software is and where it came from.

The main experiment covered 1,920 trials across three model arms and two execution harnesses. Each run occurred in a fresh, network-disabled Docker container, with file access, commands, arguments, outputs, timing, and costs recorded.

The result was not a pattern of assistants checking the evidence poorly. Most never reached that stage.

Signal availability barely changed pre-install behavior

The registered primary outcome was deliberately modest: did the assistant retrieve at least one relevant provenance signal before the installation or decline decision?

Only 9 of 1,920 registered trials met that criterion.

Pooled across signal-present conditions, 8 of 1,344 trials were positive, or 0.6%, compared with 0 of 384 controls. The planned logistic regression could not be estimated because the control condition had no positive events. The pre-registered within-project permutation fallback returned $p=0.50$.

That result should not be read as proof that provenance signals can never affect assistants. Nine positive events provide too little information for such a conclusion, and the corresponding confidence intervals permit small effects. What the experiment supports much more strongly is the descriptive magnitude: signal consumption was extremely rare under these conditions.

Even combining the available defenses did not produce a visible jump in behavior. The all-signals-present condition recorded 0 pre-decision positives in 192 trials and only one signal retrieval at any point. This condition is useful as a robustness check on the idea that richer signal availability would naturally prompt exploration; it is not evidence that supplying more provenance information causes worse behavior.

The individual condition counts were similarly sparse. The highest point estimate was the valid-attestation condition at 4 of 192 trials. Those cell-level differences are too thin to rank SBOMs, signatures, attestations, or channel declarations.

Opening a file is not verification

The study’s most operationally consequential measurement choice is separating retrieval from actual verification.

Across all 2,114 completed registered and supplementary trials, only 16 contained any provenance-signal retrieval at any time. Several of those retrievals happened after installation had already been attempted.

Actual cryptographic verification commands: 0 of 2,114.

That changes how the headline should be interpreted. A trial counted as “verification-positive” in the registered primary outcome when the assistant retrieved relevant evidence before the decision point. The term does not indicate that a signature or attestation was validated.

This also explains why the experiment cannot tell us whether assistants reliably distinguish a valid signer from the wrong signer. The study included issuer-mismatch conditions precisely to test that behavior, but the assistants almost never progressed far enough for cryptographic validity to become behaviorally relevant.

The bottleneck was upstream.

A signature can be perfectly valid or obviously wrong, but neither property affects an installation decision if the execution path never invokes the check.

Approval prompts did not create a verification workflow

One harness executed model-planned commands autonomously. The other required approval before commands and automatically granted every request.

The gated harness produced 3 registered positives in 960 trials; the autonomous harness produced 6 in 960. Meanwhile, the gated system processed 4,498 scripted approval requests across 951 trials.

Because the event count is so small, these figures should be treated descriptively rather than as evidence that autonomy improves checking. Their operational value lies elsewhere: adding an approval checkpoint did not automatically insert provenance verification into the workflow.

An approval mechanism answers, “May this command run?” A provenance mechanism needs to answer a different sequence of questions: “What artifact is about to be installed? What evidence supports its identity and origin? Does that evidence satisfy policy?”

Unless those checks are part of the gate itself, an approval layer can repeatedly authorize commands without examining software provenance.

Model price is not a substitute for the missing control

The same caution applies to model comparisons.

All nine registered positives occurred in the claude-sonnet-5 arm, while the other two registered arms produced none. A descriptive frontier-model supplement added Kimi-K2.6, gpt-5.6, and claude-fable-5 on control and all-signals-present conditions. It again found almost no pre-decision retrieval: one positive for Kimi-K2.6 and none for the other two arms.

The study cannot support a model ranking from numbers this sparse.

It does, however, challenge a specific procurement shortcut. The most expensive measured arm opened no signal in 50 completed trials. Higher inference price was not accompanied by more provenance checking in this experiment.

For a buyer, that makes provenance verification a poor feature to infer from model tier. Coding quality, privacy, latency, price, and reproducibility can remain model-selection criteria. Verification should be specified independently as an execution requirement.

Put the trust boundary before installation

The practical design implication is straightforward: move provenance consumption out of discretionary model behavior.

For an installation workflow, Cognaptus would treat the model as one component inside a governed execution path:

resolve artifact → retrieve required provenance → verify identity and claim → evaluate policy → permit or block installation → retain the evidence and decision log

The model can help determine what software is needed or explain a failed check. It does not need discretion over whether the mandatory check happens.

This design also improves auditability. Instead of asking whether an assistant “considered security,” an operator can inspect whether the expected provenance file was retrieved, which verification command ran, what identity was checked, what result was returned, and whether installation was allowed afterward.

The paper’s instrumented logging is therefore more than an evaluation technique. It illustrates the evidence an operational control should itself produce.

Keep the deployment lesson narrower than the experiment

The strongest conclusion is about the observed workflow, not every coding assistant in every installation environment.

The study covers six research-software projects, fixed prompts, modified local repositories, and isolated containers without package-index access. Most installation commands that ran returned non-zero status because dependencies were unavailable. The cryptographic material also used study-created identities rather than a live external trust infrastructure.

Only nine registered positives occurred, two projects accounted for all of them, and the planned regression was not estimable. Those facts sharply limit model, harness, and signal-class comparisons.

They do not erase the central measurement. In this controlled environment, trust information was usually available but almost never entered the decision path, and cryptographic verification never occurred.

For operators, that is enough to change the control architecture. Treat signatures, SBOMs, attestations, and official-channel declarations as inputs to an enforced verification process—not as hints that a sufficiently capable assistant may decide to inspect.

Cognaptus: Automate the Present, Incubate the Future.