Trust No One, Adjudicate Everything: When RAG Sources Disagree

TL;DR for operators

A retrieval system does not become trustworthy merely because it has documents. It becomes a system with several possible ways to be confidently wrong.

MACR treats disagreement as an adjudication problem. It estimates whether the model appears to know the answer, turns that internal position into inspectable text—or retrieves an external substitute when confidence is low—then asks specialized agents to identify contradictions and apply validated resolution rules.

The reported gains are large. Across two base models and three benchmarks, MACR improves Exact Match by roughly 20.6 to 28.6 percentage points over the strongest listed baseline. More importantly, its ablation indicates that the rule-guided multi-agent stage, not generic “think step by step” prompting, accounts for much of the improvement.

For enterprise deployment, the useful idea is not “add three agents.” It is to build a governed conflict-resolution layer with source provenance, explicit disagreement records, versioned adjudication rules, explanations, and escalation. The paper does not establish that this can be done cheaply, quickly, or reliably at production scale. Multi-agent enthusiasm remains very affordable when nobody has submitted the cloud bill.

Two documents agree with the model—and all three are wrong

Consider a familiar enterprise question: which policy applies, which office owns an account, or which version of a product specification is current?

The assistant has three sources of information. Its pretrained memory says one thing. An older document says the same thing. A newer document says something else. A conventional system may follow the context, trust its memory, count apparent agreement, or select whichever source matches a confidence threshold.

That looks like evidence handling. It is often closer to an election in which outdated facts receive multiple votes.

The paper Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference introduces MACR to address precisely this failure mode.¹ Its central correction is simple but consequential: knowledge conflict is not merely a choice between model memory and retrieved context. Both may be unreliable, several contexts may contradict one another, and the wrong context may reinforce the model’s outdated belief.

The usual binary question—

Should the model trust itself or trust the document?

—is therefore badly formed. The better question is:

What exactly conflicts, what kind of conflict is it, and which resolution rule applies to the evidence available now?

That replacement turns reliability from a preference setting into a process.

MACR turns conflict resolution into a three-stage pipeline

MACR is organized around three sequential stages:

1. Assess whether the model appears to possess usable knowledge.
2. Create an explicit basic context from internal memory or external retrieval.
3. Detect and adjudicate conflicts across that basic context and the supplied contexts.

The sequence matters. The framework does not ask one monolithic prompt to absorb contradictory passages and somehow emerge enlightened. It first makes the model’s position inspectable, then decomposes disagreement, then applies rules.

A compact representation is:

Query
  ↓
Uncertainty assessment
  ↓
High confidence → externalize internal position
Low confidence  → retrieve external anchor
  ↓
Basic context + supplied contexts
  ↓
Analyzer detects conflict
  ↓
Reasoner applies validated rule
  ↓
Answer + resolution explanation

The mechanism is more revealing than the “multi-agent” label. There are plenty of agent systems whose chief contribution is converting one opaque prompt into several opaque prompts. MACR at least assigns different epistemic jobs to different components.

Stage 1: confidence routes the inquiry; it does not settle it

The first stage generates multiple candidate answers and measures how semantically consistent they are. The authors modify a semantic-entropy method by incorporating the relevance of each answer to the question.

That addition addresses an obvious defect in consistency-based confidence. A model that answers “I don’t know” repeatedly is highly consistent and not especially useful. MACR seeks low entropy only when the generated responses are both mutually aligned and relevant to the query.

The confidence prompt also includes two pieces of metadata:

• the current date, to remind the model that its knowledge may be temporally stale;
• subject disambiguation, to reduce confusion between entities with similar names.

A validation-derived threshold then divides requests into two branches. Lower entropy indicates higher confidence; higher entropy indicates lower confidence.

This resembles confidence-based source selection, but MACR uses confidence more narrowly. It determines which candidate knowledge source becomes the basic context for later adjudication. It does not declare that source correct.

That distinction is essential. Confidence is evidence about model stability, not evidence about truth. Models can be consistently outdated, consistently miscalibrated, or consistently persuaded by a familiar falsehood. The paper’s Tesla example is built around exactly that problem.

Stage 2: make internal memory argue its case—or retrieve an anchor

When confidence is high, MACR prompts the model to externalize its internal position as text. The generated internal context includes subject attributes, relevant background, supporting details, and the model’s knowledge cutoff. The model then produces an initial answer conditioned on that text.

When confidence is low, MACR retrieves external information instead. For unstructured data, it selects the chunk with the highest semantic similarity to the query. For structured data, the paper describes selecting a relevant one-hop subgraph. The model produces an initial answer grounded in the retrieved evidence.

Either branch yields a basic context, denoted $C_0$, plus an initial answer. That pair proceeds to the conflict-resolution stage alongside the supplied contexts $C_1,\ldots,C_n$.

Operationally, this stage performs a useful transformation:

There is, however, an important boundary. “Externalizing parametric knowledge” is not a transparent extraction from model weights. It is another generated response. The text may reveal the model’s operative belief, but the paper does not establish that it is a faithful readout of the model’s internal representation.

It is an inspectable claim, not an MRI scan of the model’s soul.

That is still useful. Auditing a generated claim is easier than auditing an unspoken probability distribution. One should simply avoid granting the model’s self-description more epistemic dignity than it has earned.

Stage 3: separate observation, diagnosis, and judgment

The third stage contains MACR’s distinctive mechanism: an Observer, an Analyzer, and a Reasoner.

The Observer learns candidate adjudication rules offline

The Observer examines demonstration examples and induces rules represented as:

A temporal rule might state that when two claims refer to different dates, the claim associated with the more recent date should be preferred. Other rule types could concern source reliability or subject ambiguity.

Candidate rules are then tested on a validation set. The framework retains rules according to two measures:

• coverage: how frequently the conflict type and condition occur;
• support: how frequently applying the resolution produces the correct answer.

This is a meaningful design choice. The framework does not merely ask an LLM to invent a principle during every query. It attempts to convert recurring conflict patterns into a reusable and filtered rule base.

The Analyzer isolates the disagreement online

At inference time, the Analyzer generates an answer independently from each context. It first compares these context-specific answers at the semantic level.

When two answers conflict, it returns to the source text, extracts the precise contradictory snippets, and classifies the conflict. The resulting conflict record contains the two snippets and a conflict type.

This hierarchy—answer comparison first, evidence extraction second—reduces the task from “understand every relationship among several long documents” to “locate the passages responsible for a specific disagreement.”

That is closer to how a competent reviewer works. First notice that two submissions reach incompatible conclusions; then inspect the evidence that produced the divergence.

The Reasoner applies the rule and produces an explanation

The Reasoner retrieves rules matching the detected conflict type. It checks whether the conflicting snippets satisfy a rule’s conditions, applies the prescribed resolution, and creates a local judgment.

When no rule applies, the framework falls back to the model’s parametric reasoning and sends the unresolved case back to the Observer for future rule refinement. The final answer aggregates the available contexts and local judgments, accompanied by an explanation of the conflict and its resolution.

The feedback loop sounds adaptive, but it also creates a governance question the paper does not answer: who approves a new rule before it begins influencing later decisions?

An online rule base that quietly learns from provisional model judgments is not automatically self-improving. It may simply automate the production of precedents no one reviewed.

The Tesla case shows why agreement is not adjudication

The paper’s case study asks where Tesla’s headquarters is located.

The base model strongly associates the headquarters with Palo Alto. One context describes an earlier move to Palo Alto. Another says the headquarters has been in Austin since 2021. A vanilla in-context baseline favors Palo Alto, effectively allowing the model’s prior and the older context to reinforce one another.

MACR first externalizes the model’s belief, including its temporal scope. The Analyzer then identifies that the internal context and the older document conflict with the Austin document. The Reasoner applies a temporal update rule: when statements disagree because they describe different dates, prefer the claim associated with the more recent date.

The case is not important because Tesla headquarters trivia has finally received the procedural justice it deserves. It is important because it exposes a common failure pattern:

1. an outdated model belief appears confident;
2. an outdated document corroborates it;
3. a current source becomes a minority;
4. implicit weighting turns duplicated staleness into apparent consensus.

MACR replaces this accidental vote with an explicit temporal rule. It also produces an explanation naming the conflict, the rule, and the selected evidence.

That explanation is useful for auditability, but the case study is an illustration of mechanism, not independent proof that explanations are faithful. The paper does not test whether the generated rationale always reflects the actual causal path by which the answer was produced.

The main results show large gains on a narrow task

The main evidence compares MACR with direct answering, ordinary in-context learning, InstructRAG, TruthfulRAG, and CK-PLUG. The last baseline dynamically chooses between parametric and contextual knowledge using model confidence, making it the closest comparison to the binary source-selection paradigm the paper criticizes.

The experiments use Llama 3.1 8B and Qwen 2.5 7B on three benchmarks:

• ConflictBank, covering misinformation, temporal conflict, and subject ambiguity;
• ConFiQA, containing plausible counterfactual contexts;
• MQuAKE, requiring multi-hop answers that can contradict pretrained beliefs.

The authors report Exact Match and ROUGE-L. The table below shows MACR’s Exact Match score and its absolute gain over the strongest listed baseline for each model-dataset combination.

These are large absolute differences: 20.6 to 28.6 percentage points in Exact Match. The pattern also repeats across both base models and all three datasets, which is more persuasive than a single favorable benchmark.

The ranking of the baselines helps explain the result.

Direct answering performs poorly when the model lacks the relevant knowledge. Standard in-context learning improves when documents are supplied but remains vulnerable to misleading context. Robust RAG methods handle contextual noise more effectively but do not fully arbitrate between external evidence and internal priors. CK-PLUG can choose between those sources but still struggles when several contexts disagree.

MACR is built for the combined problem, so the advantage is coherent with its mechanism.

Still, the paper reports point estimates without confidence intervals or statistical tests. Its use of “significantly” should therefore be read as “by a large observed margin,” not as a demonstrated statistical-significance claim.

The variant-context experiment measures degradation, not a second thesis

The paper next varies the amount of contradictory context in ConflictBank while keeping only one factually correct passage. Table II and Figure 3 report results for three, four, and five contexts.

This experiment is best interpreted as a robustness or sensitivity test. It asks how performance degrades as correct evidence becomes more diluted by conflict. It does not remove a component, so calling it an ablation—something the paper briefly does—is technically generous.

MACR’s Exact Match declines from 0.549 with three contexts to 0.350 with five. Its ROUGE-L declines from 0.678 to 0.599. The system is not immune to additional contradiction; it simply degrades less severely in ROUGE-L and maintains a large advantage at the hardest setting.

At five contexts:

• MACR reaches 0.350 Exact Match, compared with 0.256 for the best baseline;
• MACR reaches 0.599 ROUGE-L, compared with 0.350 for the best baseline.

The most demanding MACR setting also exceeds the strongest baseline result in the easiest setting: 0.350 versus 0.312 in Exact Match, and 0.599 versus 0.506 in ROUGE-L.

That supports a useful but bounded claim: explicit adjudication appears more resistant to evidence dilution than asking a model to absorb noisy context directly.

The reporting contains a small inconsistency worth noting. The experimental-design prose says the number of contexts varies from two to four, while the table, figure, and subsequent analysis use three to five. The numerical results are clear; the setup description could use another editorial pass.

The component ablation says the expensive stage earns most of the improvement

The decisive diagnostic is Table III. It tests two variants on ConflictBank:

• removing Knowledge Assessment and Retrieval;
• replacing inductive multi-agent reasoning with a single-model chain-of-thought process.

Removing the assessment-and-retrieval stage reduces Exact Match by 6.9 percentage points. Replacing the rule-guided agents with generic chain-of-thought reduces it by 32.0 points.

ROUGE-L tells an even sharper story: a 2.2-point reduction without assessment and retrieval, versus a 34.9-point reduction with chain-of-thought.

On this benchmark, the structured adjudication stage contributes much more than the routing stage. The confidence mechanism helps the system choose a more useful anchor, but the main value appears to come from decomposing conflicts and applying reusable rules.

That matters for architecture decisions. A team could look at MACR and conclude that the clever part is semantic-entropy routing. The ablation suggests otherwise. The part earning its keep is the costly part: explicit comparison, conflict classification, rule matching, and synthesis.

Generic reasoning prompts are not equivalent. “Think carefully about the documents” provides no stable conflict taxonomy, no validated rule base, and no separation between detecting disagreement and deciding what to do about it.

Chain-of-thought can produce a persuasive narrative while silently changing its standard of evidence from one case to the next. Apparently, asking the model to be thoughtful does not yet constitute an adjudication policy.

The experiments support different parts of the claim

The paper contains several forms of evidence, and they should not be asked to prove the same thing.

This distinction matters because research articles often place all experimental objects next to one another and allow proximity to imply equivalence. A benchmark table is evidence of comparative performance. A case study is evidence that the proposed explanation is intelligible. Neither automatically proves that the system is ready to arbitrate a regulatory filing at 9:02 on Monday morning.

What the paper shows, what Cognaptus infers, and what remains uncertain

The business relevance is strongest where disagreements are recurrent and classifiable. Examples include policy version conflicts, customer-account ownership, product specification changes, supplier records, regulatory updates, and incident reports with competing timelines.

These environments already contain informal resolution rules:

• prefer the policy effective on the transaction date;
• prefer the designated system of record over an email attachment;
• prefer a signed amendment over an earlier contract;
• escalate when two authoritative systems disagree.

MACR’s broader lesson is to encode such rules as an inspectable layer rather than hoping the generator improvises them correctly.

An enterprise version needs governance, not merely more agents

A practical implementation would require more structure than the research prototype describes.

Preserve source identity and provenance

The Analyzer should not receive anonymous text chunks. Each claim should retain its document identifier, owner, effective date, authority level, retrieval timestamp, and version history.

A temporal rule is only as reliable as its timestamps. “More recent” is not automatically “more authoritative.” A recent chat message does not override a signed policy. Enterprise adjudication needs both recency and source hierarchy.

Treat conflict rules as controlled assets

Rules should have owners, versions, test cases, approval status, and rollback history. Coverage and support on a benchmark are useful filters, but they are not substitutes for organizational authorization.

A rule such as “prefer the latest dated document” might work for headquarters locations and fail spectacularly for contracts, where an older master agreement may remain controlling until a valid amendment is executed.

The Observer should therefore propose rules, not silently legislate them.

Preserve the conflict record

A useful output should contain more than a final answer. It should preserve:

• the claims considered;
• the sources associated with each claim;
• the identified conflict type;
• the rule applied;
• the evidence satisfying the rule;
• the resulting judgment;
• any unresolved ambiguity.

This record makes later review possible. It also prevents the explanation from being treated as decorative prose detached from the actual decision inputs.

Add abstention and escalation

MACR’s fallback asks the model to produce a provisional judgment when no rule applies. In a business system, “no applicable rule” may be precisely the point at which the model should stop deciding.

An escalation policy might depend on:

Low-impact conflicts could be resolved automatically. High-impact conflicts should be referred to an owner, particularly when the sources are similarly authoritative or the applicable rule is ambiguous.

Measure the operational economics

The framework makes repeated model calls for answer generation, pairwise comparison, snippet extraction, rule matching, resolution, and final synthesis. As the number of contexts grows, pairwise comparison can also grow approximately with $O(n^2)$ context pairs.

The paper acknowledges latency and computational cost but does not report either. Before deployment, an operator would need to measure:

• model calls per resolved request;
• token consumption;
• median and tail latency;
• conflict-detection precision and recall;
• incorrect automatic-resolution rate;
• escalation rate;
• cost per verified answer.

Without those measurements, the business case remains architectural rather than economic.

The evidence stops well before production adjudication

The paper’s limitations are not incidental. They define where the result can reasonably be used.

First, the experiments use random samples of 1,000 instances from each dataset, split into demonstration, validation, and test subsets. This is a reasonable resource-conscious research design, but it is not an exhaustive benchmark evaluation.

Second, the tasks are predominantly single-answer question answering. The formal problem assumes a correct entity related to a subject. Real business disputes are often less obliging. Two policies may both apply under different conditions. A recommendation may require balancing competing objectives rather than selecting one correct fact. A legal or financial interpretation may remain genuinely contestable.

Third, the base models are Llama 3.1 8B and Qwen 2.5 7B. The consistency of results across two models is useful, but it does not show how the framework behaves with larger models, proprietary systems, domain-specialized models, or architectures already equipped with advanced tool use.

Fourth, GPT-4o-mini supplies external knowledge in the retrieval stage. The baselines receive the same generated knowledge, which improves comparison fairness, but the setup is not a conventional retrieval evaluation over a fixed, independently audited corpus. The quality of the external anchor is partly inherited from another model.

Fifth, the conflicts are provided or constructed within benchmark settings. The system does not have to solve the entire production pipeline of identifying authoritative sources, detecting missing documents, handling access controls, resisting deliberate evidence poisoning, and determining whether an apparent contradiction is merely a scope difference.

Finally, the paper does not report latency, cost, statistical uncertainty, rule-base size, conflict-classification accuracy, or explanation-faithfulness measurements. Those omissions do not negate the accuracy results. They prevent accuracy from being mistaken for operational readiness.

The important shift is from source preference to decision procedure

RAG systems are commonly designed around a reassuring fiction: the model knows something, retrieval knows something better, and the final prompt will combine the two.

The unpleasant cases begin when neither source deserves unconditional trust.

MACR’s contribution is to make that problem explicit. It externalizes the model’s position, compares claims separately, identifies the type of disagreement, and applies a reusable resolution rule. The framework’s reported benchmark gains suggest that this procedural decomposition is more effective than choosing a preferred source or asking one model to reason more carefully.

The business lesson is not that every assistant now requires an Observer, Analyzer, and Reasoner with impressive job titles. It is that conflict resolution should become an independent, governed system capability.

When several sources disagree, the answer should not emerge from whichever statement the generator found most persuasive. It should follow a visible decision procedure: identify the conflict, evaluate the evidence, apply the authorized rule, explain the result, and escalate when the procedure runs out.

Trust is not a routing parameter. It is the output of an adjudication process.

Cognaptus: Automate the Present, Incubate the Future.