The Query Ran. The Answer Was Still Wrong: What EXYGEN Changes About Knowledge-Graph Interfaces

TL;DR for operators

A natural-language interface to a knowledge graph can look healthy while producing the wrong answer. In the experiments behind EXYGEN, several configurations generated executable SPARQL more than 97% of the time yet achieved zero correct execution results. The strongest configuration reached 97.8% executability and 41.9% relaxed exact match on execution results only after combining graph-derived metadata, explicit schema constraints, retrieved triples, and question-query examples.

For teams connecting LLMs to enterprise or scientific knowledge graphs, the implication is narrower than “fine-tuning is unnecessary.” The adaptation burden can move out of model weights and into retrievable schema descriptions, examples, namespace discipline, and execution-based validation. EXYGEN also shows that generating those structural assets does not always require processing the entire graph: sampling reduced metadata-generation time from 19 minutes 21 seconds to 13.77 seconds on GESIS and from 64 minutes 16 seconds to 11.34 seconds on OpenCitations Meta. What the study does not establish is that sampled metadata preserves downstream question-answering accuracy.

Successful execution is not evidence that the query is correct

When a team connects an LLM to a new knowledge graph, the operational target is not merely valid syntax. The generated query must reflect the graph’s schema, use the right predicates and namespaces, and return the intended result without requiring a separately fine-tuned model for every graph.

The EXYGEN paper¹ exposes how far familiar diagnostics can sit from that target. On the 513-question SciQA test set, DeepSeek-V3.1 generated executable queries for 97.4% of questions with no added context. Yet its execution-result relaxed exact match was zero. Providing only an explicit schema description pushed executability to 99.4%, while execution-result correctness remained zero.

The paper measures that final criterion using execution-result relaxed exact match, or EM_RelEx: whether running the generated query produces the reference result, with the benchmark’s relaxed matching rules. It is a stricter operational signal than asking whether the text resembles the reference query or whether the endpoint accepts it.

The distinction becomes especially visible in the smaller Qwen model:

The third row is the warning for production evaluation. A query can score 0.886 in token-level similarity and still be almost completely unusable. Here, namespace and prefix inconsistencies account for much of the failure.

What the paper directly shows: lexical resemblance, syntactic executability, and correct execution results are materially different evaluation layers.

Cognaptus inference: for production text-to-query systems, the gate closest to the business action should be execution against the target graph plus validation of the returned result. Query-string similarity is useful for diagnosis, but it should not serve as the acceptance criterion.

Fine-tuning-free moves supervision into explicit assets

EXYGEN’s adaptation strategy is to externalize knowledge that might otherwise have to be learned parametrically.

First, it extracts machine-readable descriptions of the graph: statistics and structural information represented through VoID. It then derives explicit class-level constraints using ShEx, which describes which properties and relationships are structurally valid for particular entity types. Retrieved graph triples provide local factual context. Finally, the system can retrieve previously seen natural-language question and SPARQL-query pairs as task examples.

Those examples are not a minor enhancement. Across the tested configurations that omitted question-query exemplars, EM_RelEx remained zero. DeepSeek-V3.1 using the examples alone reached 9.7%; combining them with richer graph context raised performance further. With VoID, ShEx, retrieved triples, and three retrieved question-query examples, DeepSeek reached the paper’s best result: 41.9% EM_RelEx at 97.8% executability.

The schema ablation is particularly revealing. Removing ShEx from the otherwise rich VoID + retrieved-triples + exemplar configuration drove executability to 0% for Qwen2.5-3B, 0.1% for Qwen3-30B, and 0% for DeepSeek. This is an ablation rather than a separate thesis: its purpose is to isolate what the explicit schema constraints contribute once examples are present.

That changes how “fine-tuning-free” should be interpreted operationally. EXYGEN reduces dependence on changing model parameters, but its strongest results still require graph-specific structural information and curated task examples. The supervision has become more inspectable and replaceable; it has not disappeared.

Sampling turns metadata extraction into a tractable infrastructure job

Externalizing graph structure creates another problem: someone has to generate that structure.

On large or schema-complex graphs, generating complete VoID and ShEx metadata directly from a live SPARQL endpoint can require large numbers of expensive aggregation queries. EXYGEN addresses this with parallel graph sampling designed to preserve predicate diversity. Its traversal can prune neighbors that add little predicate novelty and can choose neighbors by brute force, graph degree, or semantic similarity.

The sampling-strategy experiments are best read as a sensitivity and engineering trade-off study. On GESIS, a semantic-similarity configuration reached the highest tested predicate coverage, 57.86%, with only 0.42% of full-graph triples associated with predicates missing from the sample, but took 95.82 minutes. A two-hop brute-force configuration completed in 1.67 minutes with 30.62% predicate coverage and a 0.82% missing-triple ratio. On OpenCitations Meta, semantic sampling reached 76.64% predicate coverage in 2.41 minutes.

The end-to-end metadata-generation results make the infrastructure payoff clearer. GESIS fell from 19 minutes 21 seconds to 13.77 seconds using a 0.42% sample. OpenCitations Meta fell from 64 minutes 16 seconds to 11.34 seconds using less than 0.01% of the graph.

ORKG adds a different lesson. It contains about 1.13 million triples, far fewer than OpenCitations Meta’s 4.94 billion, yet its fine-grained schema implies roughly 18 million queries for full metadata generation. The live endpoint cannot complete that workload under its rate limits. Sampled generation completes in 43 minutes 37 seconds.

For capacity planning, raw triple count is therefore an incomplete cost signal. Classes, predicates, and their combinations can determine how many metadata queries a profiling pipeline must issue.

The deployment architecture changes before the model does

For an enterprise or scientific KG team, EXYGEN suggests a concrete redistribution of engineering effort.

Instead of assuming each new schema requires another round of model training, teams can maintain four operational assets: automatically generated structural metadata, explicit schema constraints, a small library of representative question-query pairs, and an execution-validation layer. This architecture also makes individual failure modes easier to observe. Prefix handling, schema retrieval, exemplar quality, and answer validation can be tested independently of model weights.

The likely economic benefit is not simply lower training cost. It is shorter adaptation cycles when graphs change. A schema description or exemplar library can be regenerated and inspected without producing a new checkpoint. Predicate-aware sampling can also reduce load on large production endpoints during that refresh process.

The evidence supports this as a deployment direction, not as a universal replacement for fine-tuning. The authors’ strongest fine-tuning-free result remains below the cited fine-tuned FIRESPARQL result, and the KGQA evaluation covers only SciQA.

Sampling efficiency still needs an end-task check

The largest unresolved deployment boundary sits between EXYGEN’s two contributions.

The paper shows that sampling can preserve substantial predicate coverage while cutting metadata-generation cost. It does not run the downstream KGQA experiment needed to establish that metadata produced from those samples supports the same SPARQL accuracy as metadata generated from the complete graph.

That missing comparison matters because predicate coverage and missing-triple ratio are structural proxies. They indicate how much of the graph’s relational vocabulary survives sampling; they do not measure whether the omitted structure contains exactly the information needed for a particular user question.

A production team can therefore treat sampling as a validated infrastructure optimization only after adding its own downstream test: compare execution-result accuracy using sampled metadata against the full-metadata baseline on representative workloads.

EXYGEN’s broader contribution is to make the adaptation problem more explicit. A model connected to a knowledge graph needs more than enough information to emit plausible SPARQL. It needs structural constraints, task examples, reliable namespace handling, and evaluation at the level of returned answers. Once those assets are visible, they can also be engineered, sampled, refreshed, and tested separately from the model itself.

Cognaptus: Automate the Present, Incubate the Future.