Count Coverage, Not Attempts

TL;DR for operators

More AI assessment attempts can increase the chance of resolving a security obligation, but attempt count is not assurance coverage. The operational quantity is the share of distinct assurance outcomes correctly resolved at least once.

Jithin VG and Ditto PS develop a theoretical framework for compute-bounded security assurance.¹ Its central result is that, under a fixed procedure with conditionally iid repeated attempts, long-run coverage depends on whether some outcomes are unreachable by that procedure. Mean success and pairwise correlation do not reveal that limit.

For operators, three consequences follow:

• Do not convert correlation into a presumed coverage ceiling by inserting an effective-sample-size correction into an iid success curve.
• Compare security-assessment configurations using complete workflow cost, adjudication quality, support, deadlines, and distinct timely outcomes—not tokens, reports, or inference throughput alone.
• Treat security impact as a separate causal question. Faster discovery or higher verified throughput does not establish that operational losses fell.

The framework disciplines evaluation design. It does not provide measured scaling laws, named-model comparisons, deployed benchmarks, or evidence that a cheaper model can replace a more capable one.

Another run is valuable only if it reaches something new

Suppose a security team can spend twice as much inference compute on repeated AI assessments. The immediate intuition is reasonable: more attempts should increase the chance of finding something useful.

The harder question is what the additional attempts are actually reaching.

Two assessment processes can have the same average success probability and the same pairwise correlation between repeated outcomes while approaching different long-run coverage limits. The difference is whether some assurance outcomes are merely difficult or impossible for the fixed procedure to resolve.

That distinction changes the procurement question. Instead of asking how many additional samples a budget buys, the team needs to ask which obligations those samples can eventually cover.

The paper formalizes coverage after $n$ attempts as

where $\Theta$ is the latent per-attempt success probability for a weighted randomly selected outcome. Under the paper’s conditional-iid assumptions,

A strict ceiling below one therefore comes from positive probability mass on outcomes for which the fixed procedure has zero chance of success.

This is a support problem, not merely a correlation problem.

Correlation does not tell you what repeated inference can ultimately reach

A common statistical shortcut is to summarize dependent repetitions using an effective sample size. For equicorrelated Bernoulli outcomes, the paper writes

where $c$ is pairwise correlation.

That quantity has a legitimate role: it expresses how correlation changes the variance of a sample mean. But variance-equivalent sample size does not determine the probability that at least one attempt eventually succeeds on a particular outcome.

The paper makes this distinction constructive rather than rhetorical. It presents two latent-success models with the same mean success probability and the same pairwise correlation but different asymptotic coverage. In its synthetic example with $q=c=0.1$, one model approaches coverage of 1 while another approaches $10/19\approx0.526$. These are analytic illustrations, not observed security-system measurements.

The replacement for the shortcut is therefore not a different correlation formula. It is to measure coverage directly over a declared outcome universe and to understand the distribution of task-level success probabilities.

There is also an identification limit. With a finite assessment budget, an outcome that is truly impossible can look almost identical to one with an extremely small positive success probability. A plateau in observed coverage is therefore not, by itself, evidence that the asymptotic ceiling has been reached.

A generated finding is not yet assurance evidence

The framework then moves downstream from generation to acceptance.

If $\pi$ is the base probability that a proposed conclusion is correct, $v$ is checker sensitivity, and $f$ is the false-acceptance probability, the correctness of an accepted report is

The important operational point is that checker approval and truth are separate quantities.

In the paper’s synthetic example, $\pi=0.01$, $v=0.90$, and $f=0.01$ produce accepted-report precision of about 0.476. A checker that accepts most correct proposals can therefore still produce a weak accepted evidence stream when correct proposals are rare and false acceptance is nontrivial.

Repeated checking also does not justify multiplying nominal error rates unless the checker errors are appropriately independent. Adaptive histories and shared failure modes require explicit error accounting.

For a business deploying AI-assisted assurance, generation, adjudication, and authorization should consequently remain separate stages.

Optimize distinct, correct, timely outcomes

Once verification is included, component throughput becomes an incomplete system metric.

The paper proposes optimizing the weighted value of distinct, correctly adjudicated outcomes completed by a deadline, subject to resource and false-acceptance constraints. Its service-capacity analysis makes the systems consequence explicit: speeding one stage cannot raise end-to-end throughput beyond another active bottleneck.

Inference may become faster while transfer, retrieval, duplicate work, queueing, human review, or checker latency absorbs the gain.

This motivates the paper’s notion of verified goodput: productivity should count distinct outcomes only when they are independently adjudicated as correct and delivered within the relevant deadline.

For procurement and routing decisions, Cognaptus would extend that logic into a comparison contract:

This is a business interpretation of the framework, not an empirical result reported by the paper.

Faster detection has value only if mitigation can finish in time

The final separation is between assessment productivity and protection.

Under the paper’s illustrative model with independent exponential detection and harm clocks and deterministic mitigation delay $m$,

Increasing detection rate $\gamma$ helps, but mitigation delay remains in the equation. Even arbitrarily fast detection cannot erase downstream response time.

The broader point is more general than that specific clock model: discovery speed and realized prevention are different system properties.

The paper therefore defines operational risk reduction through a counterfactual difference in losses under evaluated policies. Vulnerability counts, reports per second, or verified goodput may describe assessment productivity; none identifies causal loss reduction without an appropriate comparison design.

What this framework does not establish

The paper is theoretical. Its numerical examples are synthetic evaluations of equations and counterexamples. It reports no live deployment, calibrated population parameters, named-model benchmark, hardware comparison, empirical scaling law, or operational security trial.

Its principal coverage result also assumes a fixed evaluation universe and conditionally iid repetition given latent task-level success. Adaptive security work may violate those assumptions.

The proposed architecture—separating evidence assessment, adjudication, authorized remediation, serving, measurement, and change control—is therefore best read as an evaluation and governance design to test, not as a benchmarked reference system.

That boundary matters because the framework’s strongest contribution is methodological discipline: it identifies which quantities must remain separate before an organization can make a defensible claim about compute, coverage, substitution, throughput, or protection.

Spend compute against the assurance objective

The paper changes the unit of analysis from the model invocation to the assurance workflow.

More attempts can be worthwhile. Cheaper models can sometimes be preferable. Serving optimizations can matter. Automated adjudication can increase capacity. Faster detection can improve response.

But each claim requires evidence at the stage where the claimed benefit is supposed to appear.

For an operator allocating security-assessment resources, the practical target is therefore not maximum inference volume. It is distinct, correctly adjudicated value delivered within the deadline, under a declared resource budget and error constraint—and, when the claim concerns protection rather than assessment productivity, demonstrated reduction in operational loss.

Cognaptus: Automate the Present, Incubate the Future.