Make Verification Expensive: RedHerring Turns an Agent’s Search Budget Into a Security Surface

TL;DR for operators

An autonomous security agent can notice far more suspicious code paths than it can fully prove vulnerable during a fixed audit window. In the paper’s 300-trajectory analysis, verification consumed 73.8% of wall-clock time, compared with 7.3% for hypothesis formation; verifying one hypothesis took about 11.1 times as much time as generating one.

That imbalance creates a defensive target. RedHerring inserts safe decoy vulnerability paths that look plausible enough to investigate but are costly for an external agent to conclusively dismiss. Under matched three-hour and 300-round search budgets, the paper reports 38.7–60.4% fewer confirmed real vulnerability findings across five evaluated open-weight models. Decoys consumed 30.6–51.5% of completion tokens and an estimated 32.5–49.9% of runtime.

For repository owners, the implication is not that decoys replace patching. It is that an adversarial agent’s verification budget may be another controllable security surface. For AI-security teams, the result also argues for measuring where agent effort goes—not only how many vulnerabilities appear in the final report.

Vulnerability discovery is constrained by proof, not suspicion

A repository-scale security agent can produce many plausible leads during an audit. The harder part is determining whether a suspicious path is actually reachable, constructing an input that exercises it, executing the program, and confirming a reproducible failure.

Zhang and colleagues frame this as a resource-allocation problem in Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery.¹ Their trajectory analysis records 9,551 vulnerability hypotheses across 300 runs. Agents began verifying 7,301 of them, yet 4,576 verification attempts—62.7%—were abandoned before reaching a conclusion.

That pattern matters because it changes what “agent capability” means during a bounded security audit. Better hypothesis generation does not automatically translate into more confirmed vulnerabilities when expensive follow-up becomes the bottleneck. The agent must choose which leads deserve deeper investigation.

The paper calls this selective verification. Once that selection step becomes explicit, defensive design no longer has to operate only by removing vulnerabilities, hiding code, or interfering with exploitation. A defender can also influence which candidates consume scarce verification resources.

RedHerring inserts paths that are safe but expensive to dismiss

RedHerring operationalizes that idea by adding decoy vulnerability paths to a repository.

The decoys are not intentionally exploitable fake bugs. Each combines two components. First, a vulnerability chain derived from real CVE structures provides credible evidence that attacker-controlled input may propagate toward a dangerous operation. Second, a false bridge appears to connect that path to the dangerous sink but is constructed so that the required condition is unsatisfiable.

The defender keeps private information that makes this impossibility cheap to verify. An external agent sees only the public repository parameters and faces a computationally difficult decision problem when trying to establish that the dangerous path cannot actually execute.

This asymmetry is central to the design. A decoy that merely looks suspicious may attract a few code-reading turns and then be discarded. A decoy that is difficult to analyze but does not resemble a real vulnerability may never be selected. RedHerring needs both properties.

Its expected cost can be summarized as:

The first term captures whether the decoy earns the agent’s attention. The second captures how much effort it consumes after investigation begins. The design works only when both are large enough to compete with real hypotheses for a finite budget.

The main result is effort diversion, not model confusion

The primary experiment uses 70 vulnerable instances from 33 OSS-Fuzz projects. For each model-instance pair, the Baseline and RedHerring conditions use the same prompt, tools, container environment, wall-clock limit, round limit, and proof-of-concept verification procedure.

Across Qwen3.8-Flash, Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro, and Kimi-K3, RedHerring reduces confirmed real vulnerability findings relative to matched Baselines. The reported reduction ranges from 38.7% for GLM-5.3 to 60.4% for Qwen3.8-Flash.

The accompanying trajectory measurements make the mechanism more interpretable. Decoy investigation accounts for 30.6–51.5% of completion tokens and an estimated 32.5–49.9% of runtime across models.

That makes the result different from a generic “the model got distracted” explanation. The measured change is a reallocation of finite verification resources toward decoy paths and away from real candidates.

The paper probes that mechanism with several secondary tests:

These tests do not establish that every future agent will behave similarly. They do strengthen the narrower explanation that both selection probability and verification cost contribute to the observed benchmark effect.

Security teams may need to measure where an agent spends its audit

The paper directly shows a reduction in confirmed real findings under a particular bounded search setup. A broader operational implication follows from that mechanism.

For repository owners concerned about autonomous vulnerability discovery, the agent’s search budget could become a defensive resource alongside conventional patching and exploit mitigation. A deployment workflow might prepare certified decoy material, adapt it to repository-specific code, independently verify its safety, and require normal behavior-preservation tests before release.

The reported implementation costs are nontrivial. Preparing decoy materials for all 70 evaluation instances required about 30 hours of one-time offline work. Integrating five prepared decoys took 69.3 minutes per instance on average and increased source-code size by 13.8%. Native-test runtime overhead remained below 1%.

For teams evaluating security agents, the measurement implication may be more immediately transferable. Final vulnerability counts hide the distinction between an agent that generates better hypotheses and one that allocates verification effort more effectively. Hypothesis counts, verification abandonment, token allocation, tool-use time, and proof-of-concept confirmation reveal more about where performance is gained or lost.

The deployment boundary is narrower than the mechanism

The benchmark provides controlled evidence, but it does not establish a general-purpose repository defense.

Only five open-weight models were evaluated. Claude and GPT were excluded because alignment refusals prevented comparable experiments. Each experimental configuration was run once, limiting evidence about stochastic run-to-run variation.

The setting is also specific: OSS-Fuzz-style memory-safety discovery with defined tool access, three-hour search windows, and 300-round limits. Different vulnerability classes, larger budgets, specialized future agents, or agents trained explicitly against decoy strategies may produce different resource-allocation behavior.

Behavior preservation is another boundary. All native test suites passed, differential testing found no reported differences between paired conditions, all 350 false bridges passed the paper’s safety checks, and targeted fuzzing was used where applicable. Those checks provide empirical assurance, not a whole-program formal equivalence proof for every possible input.

The right conclusion is therefore narrower than “repositories should fill themselves with fake vulnerabilities.” The paper provides evidence that verification allocation can be manipulated without first identifying the real vulnerabilities being protected. Whether that mechanism becomes practical outside the tested environment depends on certification cost, maintenance burden, agent adaptation, and the value of the search budget being diverted.

Verification effort becomes part of the attack surface

Autonomous vulnerability discovery is often discussed as a question of whether models can recognize and exploit increasingly subtle defects. This paper adds another variable: how an agent allocates expensive verification effort after generating those possibilities.

RedHerring’s contribution is to make that allocation strategically contestable. Within the tested benchmark, safe decoys consume enough verification capacity to reduce confirmed real findings materially, and the ablations indicate that credible evidence plus sustained verification cost—not mere visual distraction—drives the result.

For defenders, that opens a security-design question beyond vulnerability removal: when an attacker has a finite autonomous search budget, which parts of that budget can the repository force it to spend?

Cognaptus: Automate the Present, Incubate the Future.