Spend Verification Where Risk Is Highest

TL;DR for operators

A long generated report can contain low-risk statements alongside claims that deserve much stronger evidence checking. Applying the strictest verification rule everywhere spends verifier capacity without distinguishing where factual failure is most likely.

FACTOR turns that problem into a routing decision. The framework estimates uncertainty for individual claims, applies progressively stricter evidence requirements as estimated risk rises, and then selects among multiple generated candidates. In the reported benchmark, FACTOR reached a FActScore of 42.3 versus 36.8 for static verification while reducing average verification calls from 194.0 to 41.9.¹

That reduction did not make the system faster. End-to-end generation took 51.0 seconds for FACTOR versus 22.8 seconds for static verification because uncertainty estimation and additional candidate sampling add their own inference cost.

For teams building research assistants, report generators, or other accuracy-sensitive systems, the design implication is to treat verification intensity as a routing parameter. The paper’s particular thresholds are useful as an example, not a production default: its own ablations show that changing those boundaries materially changes factuality.

Every claim does not deserve the same verification budget

Consider a generated research brief containing a basic date, a numerical claim, a causal assertion, and a description of a person’s career. A verification pipeline can apply one support requirement to all four statements, but that policy assumes their probability and consequence of being wrong are similar enough to justify identical treatment.

The paper starts from a different premise: factual risk varies within the same output. FACTOR—FACTuality-Oriented Risk-aware Verification—uses that variation to allocate verification effort rather than treating verification as one uniform post-processing layer.¹

The workflow first retrieves evidence using an equal-weight combination of BM25 lexical retrieval and dense semantic retrieval, with entity-aware filtering intended to reduce cross-entity contamination. It then generates four candidate biographies from the retrieved evidence.

Only after generation does the system estimate which claims warrant stricter scrutiny. Its main uncertainty score averages two signals:

Token entropy measures how diffuse the model’s local token probabilities are. Semantic consistency measures whether additional sampled continuations remain meaningfully aligned. The first signal captures local uncertainty; the second captures instability in what the model is saying.

That score determines how demanding the evidence check becomes.

The verification model asks whether retrieved evidence supports each generated claim. High-uncertainty claims face an additional requirement: support must come from multiple retrieved passages.

The resulting architecture is therefore more than RAG with another filter attached. Retrieval supplies evidence; uncertainty decides how aggressively that evidence should be interrogated; verification scores support; and a final re-ranking stage chooses among candidate outputs using 60% factuality weight and 40% perplexity-derived fluency.

Selective verification beats the uniform policy in this benchmark

The main comparison uses 50 entities from the FActScore biography benchmark. The generator, embedding model, verifier, Wikipedia evidence snapshot, generation parameters, and four-candidate setting are held fixed across methods.

The result is a clear progression. Zero-shot generation reports a mean FActScore of 6.5. Standard RAG reaches 19.7. Static verification reaches 36.8. FACTOR reaches 42.3.

Hallucination rates move in the opposite direction: 93.5 for zero-shot generation, 80.3 for standard RAG, 63.2 for static verification, and 57.7 for FACTOR. The paired Wilcoxon comparison between static verification and FACTOR is reported at $p=0.040$.

Those numbers support two distinct conclusions. First, retrieval by itself leaves substantial unsupported content in this setting. Second, once verification is already present, changing where verification effort is concentrated can still improve the result.

The cost comparison strengthens that interpretation. Static verification averages 194.0 verification calls; FACTOR averages 41.9. The paper therefore provides evidence that selective verification can outperform a uniform verification policy while invoking the verifier much less often.

Fluency does not show an observed perplexity penalty relative to static verification: both report mean perplexity of 5.5. That result is narrower than a human-quality claim, however. Perplexity is the only reported fluency proxy; the study does not include human readability or fluency evaluation.

The ablations show what must be calibrated

The paper’s ablations are important because they test the mechanism rather than offering a second headline benchmark.

The uncertainty-signal ablation asks whether entropy and semantic consistency contribute equally useful routing information. Consistency-only routing produces the highest reported FActScore in that analysis at 50.9 with 36.7 verification calls. Entropy-only reaches 42.9 with 39.2 calls, while the equal-weight configuration reaches 46.4 with 37.1 calls. The authors retain equal weighting as a balance between factuality and robustness rather than selecting the numerically highest consistency-only configuration.

That makes semantic disagreement across samples especially interesting as a routing signal, but it does not establish a universal weighting rule.

The threshold-sensitivity test makes the calibration problem more explicit. The reported default boundaries of 0.30 and 0.70 produce a FActScore of 40.9 in that ablation. Moving the boundaries to 0.40 and 0.80 raises the reported score to 47.9, with verification calls increasing from 37.1 to 38.6. Very tight 0.10/0.40 boundaries fall to 36.9.

These are not portable constants. They determine which claims receive stronger checks, so changing them changes the verification policy itself.

Claim segmentation behaves similarly. Sentence-level verification reports FActScore 46.3 with 34.4 verification calls, versus 41.5 and 35.9 calls for atomic decomposition. Breaking text into smaller facts therefore did not automatically make verification easier. In this experiment, retaining sentence context appears to help the entailment model judge support.

One reporting detail deserves care: the default or equal-weight FActScore values in the ablation tables do not exactly reproduce the main FACTOR score of 42.3, and the source package records no explicit reconciliation. The ablations are best read comparatively within their respective tests rather than as exact reproductions of the headline run.

Verification-call efficiency and response latency are separate objectives

The most consequential operational mismatch appears in timing.

FACTOR reduces average verification calls from 194.0 to 41.9 relative to static verification. Yet generation time rises from 22.8 seconds to 51.0 seconds.

The reason is architectural. FACTOR saves work inside the verification stage, but it introduces work elsewhere: additional sampled continuations for semantic consistency, uncertainty computation, and multiple candidate generations all consume inference time.

For an engineering team, “verification cost” therefore needs at least two measurements. One is resource use by the verifier itself. The other is user-visible or pipeline-level latency. Optimizing the first does not guarantee improvement in the second.

This distinction changes where the approach is easiest to justify. An asynchronous research report can tolerate tens of additional seconds if the factuality gain is valuable. An interactive assistant with a tight response-time target may reject the same architecture even when verifier usage falls sharply.

For deployment, make verification a routing policy

The paper directly shows a benchmark result: claim-risk routing improved factuality over the tested static verification policy while using fewer verification calls.

Cognaptus would extend that result into a deployment design rule rather than copy the paper’s thresholds. In research assistants, reporting systems, and other accuracy-sensitive workflows, an uncertainty score can become one input to a broader routing policy: lighter automated checks for lower-risk statements, stronger multi-source verification for higher-risk claims, and human review where the business consequence of error justifies escalation.

The affected decision is not simply whether verification is enabled. It is which claim receives which level of evidence requirement under which service constraint.

Three parameters should be calibrated separately: factuality, computational verification cost, and end-to-end latency. The paper demonstrates that they do not move together.

Where the evidence stops

The evidence remains medium-strength rather than deployment-complete. The main test covers 50 biography entities, one Phi-2 generator, one embedding model, one NLI verifier, and one March 2022 Wikipedia evidence snapshot. There is no cross-model or cross-domain replication.

Threshold sensitivity further means that production performance will depend on how uncertainty scores behave in the target system. A boundary calibrated for biographies generated by this model stack should not be assumed to transfer to financial research, technical reporting, customer support, or a different generator.

FACTOR nevertheless isolates a design question that deserves more attention in factual-generation systems. Once retrieval and verification already exist, the next improvement may come from deciding where verification effort belongs. The paper shows that this allocation can materially change factuality and verifier usage. It also shows why the surrounding inference machinery must be measured before anyone calls the resulting system cheaper or faster.

Cognaptus: Automate the Present, Incubate the Future.