The Graph Is Not the Guardrail: Route Retrieval by Failure Mode

TL;DR for operators

A retrieval system may have several ways to answer the same analyst request: search semantically similar text, follow explicit relationships in a knowledge graph, repair a failed graph query, or combine graph and text evidence. The operational question is not which technique has the highest average score. It is which path fails acceptably for the workload in front of it.

Hamzic and colleagues test that question in a controlled cyber threat intelligence benchmark covering 3,300 question instances across five LLMs.¹ Against semantic RAG, plain graph-only retrieval improves the overall LLM-as-a-Judge score by just 1.34 points. Adding explicit critique and repair raises the gain to 10.68 points, while hybrid graph-plus-text retrieval reaches 11.68 points.

The difference comes from failure handling. Graph grounding is strong when a question maps cleanly onto known entities and relationships. It becomes brittle when the system mistranslates the request into a graph query, the ontology does not represent the needed fact, or the information is absent. Agentic repair reduces query-generation failures. Hybrid retrieval adds another evidence path when the graph is insufficient.

For CTI assistant operators, this supports a routing approach: use graph grounding where relational structure matters, constrain repair loops, fall back when schema coverage is doubtful, and evaluate abstention and latency alongside answer quality. The benchmark does not establish that the same ordering will hold outside its CTI corpus and controlled ontology.

The graph helps when the question fits the graph

Some analyst questions are difficult for conventional semantic retrieval because the answer depends on several facts that must be connected. Retrieving individually relevant text chunks does not guarantee that all required pieces appear together.

A knowledge graph changes that retrieval problem. Instead of asking only which passages resemble the question, the system can traverse explicit relationships among entities. In the benchmark, this advantage is clearest on factual and relational questions.

For multi-hop questions, relative to semantic RAG, the LLM-as-a-Judge improvements were:

The pattern is even stronger for simple fact questions: AGRAG improves by 22.81 points, HRAG by 18.22, and GRAG by 13.04.

These are main benchmark results, not isolated ablations. They support a bounded claim: when required evidence is represented as traversable graph relationships, explicit relational retrieval can outperform relying on semantic similarity alone.

But that condition matters. The same graph-only system that improves multi-hop retrieval performs 17.67 points worse than semantic RAG on guided analyst-style questions. AGRAG remains 8.74 points worse, while HRAG is 4.32 points better.

Guided questions require broader synthesis than a controlled relational lookup. A graph can expose relationships precisely without containing all the contextual information needed to answer the analyst’s actual question.

That is why the benchmark does not support replacing vector retrieval with GraphRAG across an entire workload.

Graph retrieval creates a new failure point before retrieval even begins

A graph system must translate a natural-language request into an executable query. In this study, that means generating Cypher for Neo4j.

This translation step creates a failure mode that semantic RAG does not have. A question can be understandable to the language model yet still generate malformed Cypher, target the wrong relation, or request a property the graph schema does not contain.

Plain GRAG therefore shows a bimodal profile. When its graph query works, it produces many excellent answers. When the translation or schema assumption fails, performance can collapse. Across all 3,300 paired instances, GRAG gains only 1.34 judge-score points over RAG, with a very small effect size of 0.05.

Model choice also changes this risk materially. In the paper’s model-sensitivity analysis, the GRAG near-failure rate was 4% for Kimi K2 Thinking, but 36% for Mistral Small 24B and 41% for Mixtral 8x7B. The retrieval architecture is therefore partly dependent on the model responsible for translating requests into graph operations.

Schema coverage produces a second problem. A graph query can be executable and return structured output while the graph still lacks the information required by the user’s question. The paper describes this as a route to structural hallucination: formal structure can make insufficient evidence appear better grounded than it is.

One representative failure makes the operational consequence visible. GRAG remained in a correction process for 2,348 seconds while searching for a graph property absent from the schema. Missing information became a latency problem instead of an abstention decision.

For deployed systems, query success and evidence sufficiency therefore need to be treated as separate checks.

Agentic repair and hybrid retrieval protect against different failures

AGRAG inserts an explicit critique-and-refinement stage around graph-query generation. Rather than repeatedly trusting the first translation, the system assesses and revises problematic Cypher before answer synthesis.

The benchmark indicates that this mechanism materially changes the failure distribution. AGRAG’s overall judge-score gain over RAG is +10.68, compared with +1.34 for plain GRAG. Its near-zero-score rate is 11.7%, versus roughly 26.7% for GRAG.

HRAG takes a different approach. It combines graph execution with semantic text retrieval, so a graph failure or schema gap does not necessarily remove the underlying documents from consideration. Its overall gain reaches +11.68, and its near-zero-score rate falls to 4.8%.

The hallucination results show the same broad pattern under the paper’s faithfulness-based definition: 45.8% for RAG, 34.1% for GRAG, 17.4% for AGRAG, and 12.4% for HRAG. Those aggregate values should not be read as universal safety rankings, however, because the ordering changes by question category.

Unanswerable questions are especially revealing. Under the paper’s separate binary abstention measure, HRAG refuses correctly 76% of the time. GRAG reaches 44%. RAG and AGRAG record 0% correct refusal and attempt an answer in every evaluated unanswerable case.

The result is a useful reminder that improving ordinary answer quality and learning when not to answer are separate system properties.

Route by failure cost, not by one benchmark average

The architectures also differ substantially in runtime.

Semantic RAG is fastest, with per-model mean runtimes ranging from 0.87 to 4.03 seconds. AGRAG ranges from 4.03 to 11.18 seconds and is relatively stable. HRAG ranges from 4.60 to 46.93 seconds. Plain GRAG is both slower and much less predictable, reaching per-model means as high as 127.53 seconds.

This creates a concrete deployment decision.

Cognaptus inference: for a CTI assistant serving mixed workloads, retrieval architecture is better treated as a routing policy than as a permanent backend choice.

The paper’s adversarial analysis reinforces this routing view. Different systems fail on different query characteristics: semantic RAG is vulnerable to precise fact and multi-hop failures; graph-only retrieval is highly exposed to schema-evasion and long guided questions; AGRAG remains weaker on some aggregate multi-hop requests; HRAG’s residual weaknesses concentrate more around unanswerable and aggregate cases.

That diversity is potentially useful.

An oracle that always chose the better answer between AGRAG and HRAG would reduce near-failure incidence from 4.8% for HRAG alone to 0.9%. Selecting the best of all four systems drives the benchmark failure rate to zero.

This is an exploratory upper bound, not a production routing result. The unresolved problem is selecting the better answer without access to the benchmark’s oracle. Still, the result shows why disagreement detection, fallback chains, or learned routing deserve attention: the systems do not fail on exactly the same questions.

The boundary is the benchmark, not every retrieval deployment

The comparative evidence is unusually detailed: 3,300 paired question instances, five heterogeneous LLMs, ten runs per model, several question categories, bootstrap intervals, effect sizes, runtime analysis, abstention measures, and explicit failure inspection.

Its external boundary is equally specific.

The corpus contains one cyber threat intelligence dataset. The knowledge graph uses one deliberately controlled ontology. Most questions are generated rather than collected from analysts in operational settings. No human analyst study tests whether the systems improve real incident-response decisions.

Ontology design could also change the outcome. A denser or broader graph could reduce schema failures, while a poorer extraction pipeline could introduce different ones.

The strongest supported conclusion is therefore architectural rather than universal: graph grounding can materially improve structured retrieval, but reliability depends on the controls surrounding the graph. Query repair, evidence fallback, abstention checks, bounded retries, and routing are part of the retrieval design rather than cleanup work after retrieval is chosen.

For teams building CTI assistants, the decision is no longer simply whether to add a knowledge graph. It is which questions should enter the graph path, what should happen when that path is uncertain, and how much latency and redundancy the application can justify to keep a retrieval failure from becoming an analyst-facing answer.

Cognaptus: Automate the Present, Incubate the Future.