Same Count, Different Bugs: Build LLM Security Scans as an Evidence Funnel

TL;DR for operators

Two runs of an LLM security scanner can return almost the same number of findings while disagreeing substantially about which code locations are vulnerable. That makes finding count a weak reproducibility metric and a risky basis for remediation decisions.

Bugstone-E2E, introduced in The History Is the Detector: Executing CVE Patch History, End-to-End,¹ offers a broader operating model. It turns verified CVE fixing commits into reusable detection skills, uses deterministic machinery to shrink the search space before asking an LLM for semantic judgments, and sends stronger claims through separate runtime validation.

The practical design principle is not “use an LLM to find vulnerabilities.” It is to assign different evidence burdens to different stages. Cheap deterministic checks establish where to look. Bounded model judgments decide what deserves attention. Repeated or corroborating judgments identify unstable cases. Runtime execution determines which findings support stronger security claims.

That staging matters because cheaper detection can simply move the bottleneck downstream. In the paper’s 14-project deployment, 2,933 deduplicated static findings entered the registry, but only 644 ultimately obtained runtime evidence. Organizations adopting this architecture need validation capacity, not merely scanner throughput.

Similar totals can hide different vulnerabilities

Suppose a security team scans an unchanged repository twice. One run reports 130 findings; another reports 140. The natural conclusion is that the scanner is reasonably stable.

The paper’s controlled Pillow experiment shows why that conclusion can fail.

Ten model-based verification runs received the same 8,115 candidates. Each produced between 125 and 147 final finding locations. Yet combining the runs produced 243 distinct locations, and only 56 appeared in every run.

The totals were stable. The identities were not.

That distinction changes how reproducibility should be measured. Across all candidates, 93.1% received the same verdict in every run when UNKNOWN variation was included, so the model was not behaving randomly across the entire workload. But 509 candidates showed direct BUGGY-versus-FALSE_POSITIVE disagreement. The operational instability was concentrated in a smaller decision boundary that materially changed which locations would enter a security workflow.

Repeated execution recovered more of that tail, with diminishing returns. Three runs captured 75.8% of the ten-run union on average; five captured 85.7%. Findings supported by multiple BUGGY judgments within a run were also substantially more stable across runs than findings resting on a single judgment.

For an operator, this supports a concrete policy: treat corroboration as evidence. A lone semantic verdict and a repeatedly supported verdict should not automatically receive the same escalation priority.

The scanner starts by turning vulnerability history into executable knowledge

Bugstone-E2E does not begin by giving an agent an entire repository and asking it to “find bugs.” Its first move happens earlier: convert historical fixes into reusable detection knowledge.

The system surveys 19,325 high-severity CVEs published from 2022 through 2026. It resolves 5,902 to identifiable open-source repositories and verifies 2,710 fixing commits. After validating whether those patches contain reusable evidence, it retains 2,662 CVE cases.

From those cases, the pipeline extracts elements such as the vulnerable operation, a source-code anchor that can be searched for, the unsafe condition, and the semantics of the fix. Candidates that merely point at a newly added sanitizer or guard, lack sufficient evidence, or are too specific to one patch are rejected.

The surviving material becomes 1,757 initial rules across 266 CWE categories, later consolidated into 1,033 production rules across 56 CWE families and packaged into 172 language-specific skills. Crucially, those rules retain provenance back to the CVEs that motivated them.

The business value is not simply a larger rule library. Vulnerability history becomes a persistent detection asset rather than context that an analyst or agent must rediscover for every scan. Provenance also gives reviewers a path from a new finding back to the historical fixes that justify the detection condition.

The boundary is structural: this approach is strongest where recurring vulnerabilities can be anchored near identifiable operations or APIs. One-off design failures, race conditions, global protocol state, configuration errors, and other stateful or API-independent problems may fall outside the rule base entirely.

Spend deterministic computation before semantic reasoning

Once the skills exist, Bugstone-E2E orders work by cost.

Tree-sitter first enumerates source locations matching the rule anchors without requiring a project build. Conservative deterministic filters then remove candidates that can be shown to be benign or redundant before model reasoning begins.

Across 15 scanner-efficiency targets, that filtering reduces roughly 745,000 rule-matched candidates to about 293,000—a 60.6% reduction before LLM-based semantic verification.

This is more than an optimization detail. It defines where model intelligence is economically justified.

The remaining candidates are grouped into bounded tasks that share a rule and source file. The model examines callers, data flow, guards, sanitization, and nearby context rather than searching an unrestricted repository from scratch. The paper’s model-cost experiment is limited to this bounded verification task, but within that setting repeated smaller-model runs could achieve broad finding-set coverage at budgets comparable to or below a single expensive frontier-model run.

Cognaptus inference: organizations designing agentic security systems should treat task narrowing as part of model selection. The question is not only which model performs best in isolation, but how much ambiguity the surrounding deterministic system leaves for the model to resolve.

A model verdict is an evidence state, not the end of the workflow

After deterministic filtering, the model-based stage—Phase B in the paper—produces structured static judgments. But Bugstone-E2E does not treat a BUGGY label as equivalent to observed vulnerable behavior.

A separate runtime stage attempts to construct executions and obtain security-relevant signals such as sanitizer violations, crashes, controlled-sink reachability, or proof-of-concept behavior.

That separation materially changes the interpretation of deployment results. Across 14 third-party projects, the system recorded 2,933 deduplicated Phase B findings. Runtime-oriented Phase C adjudicated 2,125 of them. Of those, 644 obtained runtime evidence, 1,099 were rejected as not a bug or not exploitable, and 382 retained static-reachability arguments without a runtime signal. Another 808 findings remained untested.

The last category matters. Untested is a capacity or reproducibility state, not a negative security result.

This creates a cleaner governance vocabulary than a single vulnerability label: static evidence, runtime evidence, exploit confirmation, rejection, and untested backlog can drive different remediation and escalation decisions.

It also exposes a resource constraint. Improving the economics of static detection may increase the queue requiring builds, harnesses, inputs, instrumentation, and execution environments. Runtime validation can become the new scarce resource.

Remediation gets its own execution check

Bugstone-E2E extends the same evidentiary logic into a small remediation prototype.

For 23 reproducible proof-of-concept findings across three targets, the system generated compact candidate patches with a median of eight changed lines. All 23 passed its two-sided differential test.

The test is deliberately harder than “the exploit stopped after applying the patch.” The proof of concept must stop reproducing with the patch installed and then reproduce again when that patch is reverted. The second half helps detect cases where the apparent success came from a broken environment rather than the code change itself.

This is a sensible control for agent-generated remediation, but the experiment is preliminary. It does not establish that the patches completely remove each root cause, introduce no regressions, or would be accepted by maintainers. The reported validation also relies on runtime output transcribed by the remediation worker rather than independent re-execution by the authors.

Build policy around evidence transitions, not one-shot confidence

The strongest operational lesson from Bugstone-E2E is the architecture of the decision process.

Historical fixes define reusable detection knowledge. Deterministic mechanisms search broadly and cheaply. Models handle the narrower semantic cases that remain. Corroboration and reruns reveal which model judgments are stable. Runtime validation supports stronger claims. Patch acceptance requires another execution-based check.

That sequence provides a practical answer to the reproducibility problem exposed by the Pillow experiment. When similar scan totals can conceal different findings, the organization cannot govern the workflow by counting alerts alone. It needs rules for how evidence changes status.

For security leaders, that means deciding in advance which findings can remain static leads, which require repeated support, which justify expensive execution, and what level of proof is required before remediation enters a release process.

The paper supports that staged design across the reported environments. It does not show that CVE-derived rules cover the full vulnerability space, that one agentic harness is generally superior, or that automated patches are generally correct. Those boundaries do not weaken the core mechanism. They define where human judgment, broader analysis, and independent validation still have to enter the system.

Cognaptus: Automate the Present, Incubate the Future.