When a Knowledge Graph Rejects Correct Information

TL;DR for operators

A knowledge system can reject correct information and still report excellent accuracy. In a climate-science study, ontology filtering removed 11,050 extracted assertions, although 93.4% were judged textually correct. The complete pipeline ultimately achieved 98.96% evaluator-judged correctness, but retained only 23,327 of 39,964 initial assertions.

The operational lesson is to separate three admission requirements: whether information fits the data model, whether its cited evidence can be located, and whether that evidence supports the assertion. Each check solves a different problem. Enterprise teams should measure factual correctness, provenance failures, and valid-claim rejection separately, especially when a conservative knowledge base may omit information that users need.

A verification filter rejects 11,050 assertions—mostly correct ones

Consider an organization turning thousands of research reports into a searchable knowledge base. An AI system extracts statements from documents, checks them, and adds approved information to a structured database. The organization wants fewer unsupported statements, so rejecting questionable output appears to be a sensible quality-control strategy.

The difficulty appears when a filter rejects information for reasons unrelated to factual accuracy.

In Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications, Aayush Acharya and coauthors¹ report that a rule-based verification stage removes 11,050 extracted statements. Yet 93.4% of those rejected statements are judged textually correct. The problem is that they cannot be represented within the system’s permitted data structure (Section 4.3; Appendix A.8).

This matters because a knowledge base is evaluated not only by whether its retained claims are correct, but also by whether it contains the information required for subsequent decisions. A restrictive filter may improve structural consistency while discarding relevant scientific findings.

The paper makes that tradeoff unusually visible.

Three checks answer three different questions

The researchers construct a climate-science knowledge graph using structured metadata and information extracted from 2,000 scientific publications.

A knowledge graph represents assertions as relationships between entities. For example, a climate model may produce a particular variable. That statement becomes a structured triple: subject, relationship, and object. The framework also attaches a supporting passage from the publication.

The graph begins with a deterministic foundation assembled from ESGF dataset metadata, CMIP6 controlled vocabularies, NASA GCMD concepts, and GeoNames identifiers. Gemini-2.5-pro extracts additional assertions and evidence passages from scientific PDFs. Three verification stages then determine which assertions enter the graph.

The first two checks are deterministic. Ontology conformance tests candidates against 35 permitted type-relation combinations and additional semantic guards. Evidence validation uses fuzzy text matching against parsed PDFs. The third check asks Gemini-2.5-flash to judge semantic support.

These distinctions are consequential. A claim can be factually accurate but incompatible with the ontology. A quotation can exist in a document without supporting the relationship attributed to it. And an assertion may be semantically plausible while lacking retrievable evidence.

The pipeline also resolves accepted entities to canonical identifiers, using strict matching for standardized names and embedding-based retrieval with LLM fallback for contextual concepts. This helps connect publication findings to existing climate metadata without indiscriminately merging similar-looking scientific identifiers.

The ablations reveal which checks actually improve correctness

The study evaluates all eight combinations of ontology, evidence, and semantic verification on the same 39,964 initial candidate triples. This ablation is more informative than the final correctness score because it isolates what changes when individual checks are omitted.

Source: Section 4.3, Tables 3–4. Correctness is judged using Gemini-2.5-flash.

Semantic validation produces the clearest improvement in the reported correctness measure. Adding it after ontology and evidence checks increases correctness from 94.72% to 98.96%, a gain of 4.24 percentage points.

Evidence verification behaves differently. Without that stage, the system retains 845 additional triples whose supporting passages cannot be matched to the parsed source. Yet reported correctness remains 98.96%.

The evidence check therefore contributes something the correctness metric does not adequately capture: whether an assertion can be traced back to located source material.

Ontology filtering similarly enforces representability. Its substantial rejection of textually correct statements demonstrates a coverage cost, not merely better error detection.

For an operational knowledge system, these are distinct quality objectives. A single accuracy metric cannot tell managers which objective has improved or deteriorated.

A 98.96% score does not mean 98.96% knowledge coverage

The complete pipeline retains approximately 58% of its initial candidate assertions. Its reported 98.96% correctness measures the proportion of retained triples judged valid, not the proportion of all relevant scientific relationships successfully captured.

The constructed graph is substantial: 133,990 nodes and 755,308 relations, including 73,283 nodes and 298,747 relations added through publication enrichment. Those totals also incorporate deterministic initialization and publication-related structure; they should not be interpreted as entirely LLM-extracted scientific discoveries.

Two comparisons help establish the method’s performance under specific conditions.

Against climate-adapted KARMA on 1,007 publications, the framework produces 17,141 triples at 99.067% correctness, compared with 3,820 at 92.277% for KARMA. Against ClimateIE on its 25-paper corpus, using the ClimateIE ontology, the proposed framework achieves 92.936% correctness against 75.004%, but produces only 637 triples compared with ClimateIE’s 5,729.

The second comparison is particularly instructive. Higher correctness accompanies a much smaller output set. Without a comprehensive relation-level recall evaluation, these figures cannot establish which system captures more of the useful knowledge in those publications.

An external entity-level evaluation provides a different view. Among 1,133 gold entity annotations recoverable from parsed text across 139 publications, the pipeline extracts 1,023, yielding 90.29% recall. However, only 50.78% of all externally curated gold entities appear in the parsed documents. The reported recall is conditional on textual availability and says nothing definitive about complete relation extraction.

Independent checks help, but evaluation dependence remains

The researchers also compare Gemini-2.5-flash judgments with three human annotators on 1,000 raw extracted triples. Pairwise Gemini-human Cohen’s kappa ranges from 0.640 to 0.700, compared with human-human values of 0.616 to 0.689. Among 631 full-pipeline-retained triples in that annotated subset, the human majority accepts 625, or approximately 99%.

This lends credibility to the reported precision within the evaluated climate task.

However, the main full-corpus correctness evaluator is also Gemini-2.5-flash, the model used for semantic validation. Agreement between a validator and a subsequent evaluator using the same model is not fully independent verification.

Document parsing introduces another boundary. Evidence-span mismatch rates range from 6.137% to 7.741% across tested matching thresholds when table and figure interpretations are included, but only 0.626% to 1.443% when excluded (Appendix A.21). Some apparent provenance failures therefore reflect differences in document representation, not necessarily invented evidence.

Enterprise knowledge systems need separate admission metrics

For organizations building knowledge bases from contracts, policies, research reports, or technical documentation, Cognaptus draws an architectural implication from these findings.

An extraction pipeline should record not just whether an assertion was accepted, but why. Schema incompatibility, missing source evidence, and insufficient semantic support should remain distinguishable outcomes.

That separation enables different operational responses. A high rate of schema rejection may justify revisiting the organization’s data model. Unmatched evidence may indicate parsing problems or fabricated passages. Semantic rejection suggests unsupported interpretations or insufficient context. Treating all three as generic extraction failures obscures the intervention each requires.

Organizations can also track rejected-but-plausible assertions as a separate coverage measure. This is a proposed governance practice, not a downstream benefit demonstrated by the paper. It would help prevent improved retained precision from concealing a deteriorating ability to answer users’ questions.

The implementation suggests that this separation need not make every verification step expensive. The authors estimate approximately USD 400 in extraction API costs for 2,000 publications, against approximately USD 3 for semantic verification of 27,900 candidates (Appendix A.11). These are reported API estimates, not total operating costs: local GPU processing, PDF parsing, infrastructure, and engineering expenses are excluded.

The remaining uncertainty is operational transfer. Climate science benefits from authoritative controlled vocabularies and standardized identifiers. Organizations with inconsistent entity definitions, evolving policies, or ambiguous relationships should not assume equivalent correctness, coverage, or costs. Nor does the paper establish comprehensive improvements in downstream question-answering performance.

A better knowledge base needs more than a cleaner score

The central contribution of this work is a disciplined separation of verification responsibilities.

Structural checks determine what a knowledge system can represent. Evidence checks determine what it can trace. Semantic checks determine whether the cited material supports what the system claims. The study shows that these decisions can have markedly different effects on retained correctness and information coverage.

For enterprise deployment, success should therefore mean more than maximizing the percentage of accepted statements judged correct. It also means knowing which valid information was excluded, whether retained assertions remain auditable, and whether the resulting knowledge base supports the decisions it was built to serve.

Cognaptus: Automate the Present, Incubate the Future.