Relevance Is Not Verification: Choosing an ESG RAG Model

TL;DR for operators

A sustainability analytics team can receive an AI-generated answer that addresses its question and follows a retrieved corporate disclosure, yet still lacks the information needed for accurate reporting.

A benchmark of seven open-source language models for ESG question answering confirms that these qualities must be evaluated separately.¹ Answer relevancy ranged from 0.760 to 0.881, while factual correctness remained between 0.387 and 0.449. Different models led the three principal generation metrics.

For organizations selecting locally deployable models, the study supports three practical conclusions:

• Choose according to the task. Source grounding, question responsiveness, and agreement with verified facts are distinct requirements.
• Evaluate retrieval alongside generation. Supplying only one document chunk may leave important supporting evidence unavailable to the model.
• Validate before consequential use. The benchmark provides comparative screening evidence, not human-verified production accuracy.

The immediate priority is to establish which failures an ESG workflow can tolerate and which require independent verification.

A relevant answer can still miss the required fact

Consider an analyst comparing greenhouse-gas emissions across companies. The question may require a specific reporting year, emissions scope, organizational boundary, and measurement unit.

An AI assistant might identify the correct topic and produce a coherent response from a sustainability report. Yet the answer could omit a qualification or fail to include information contained elsewhere in the document.

For an exploratory search, this response may be helpful. For a reporting workflow that transfers extracted values into a controlled dataset, incomplete information creates a different operational risk.

This is the problem examined by Motaz Saad and colleagues in Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain. The researchers compare seven open-source generators under a shared retrieval-augmented generation (RAG) pipeline.

Their results show why selecting the model with the most convincing or relevant answers is insufficient when factual agreement is a separate requirement.

Three quality measures produce three different leaders

The study evaluates generation quality using RAGAS, a framework that measures different aspects of document-based question answering.

Three measures are particularly relevant to model selection:

Faithfulness evaluates whether claims in an answer are supported by retrieved context. An answer can faithfully reflect a passage while remaining incomplete relative to the user’s question.

Answer relevancy measures how closely a response addresses the question. A relevant answer may still contain unsupported or incorrect details.

Factual correctness measures agreement between generated claims and a reference answer through an F1-style comparison. It is not a calibrated probability that the response is correct.

The reported scores illustrate these distinctions.

Source: Table III, Section IV-A. Values are aggregate RAGAS scores on 100 synthetic questions, not human-verified accuracy rates.

GLM-4.7-Flash leads faithfulness at 0.822 but records the lowest factual-correctness score, 0.387. Ministral-3-8B has the highest answer relevancy, 0.881, but the lowest faithfulness, 0.607. Qwen3-4B leads factual correctness at 0.449 without leading either of the other two measures.

The largest observed generation-score spread is in faithfulness: 0.215 between the highest and lowest models. Answer relevancy spans 0.121, while factual correctness spans just 0.062.

Gemma3-4B offers a relatively balanced profile, with 0.776 faithfulness, 0.877 answer relevancy, and 0.427 factual correctness. Gemma4-e2b, the smallest evaluated model, achieves 0.784 faithfulness and 0.423 factual correctness, suggesting that nominal model size alone is a poor selection rule for this particular benchmark.

These are descriptive comparisons, not statistically established superiority claims. No confidence intervals or significance tests are reported.

More consequentially, every model scores below 0.45 on factual correctness, even though answer relevancy is substantially higher. That pattern cautions against treating a convincing, contextually appropriate response as evidence of reliable factual extraction.

The benchmark controls the pipeline, but retrieval remains constrained

The researchers assemble 498 ESG reports from EU-listed companies, published between 2010 and 2024. The corpus contains narrative disclosures alongside quantitative information presented in heterogeneous document structures.

Their PDF-to-Markdown processing and chunking workflow produces 258,590 text chunks. From 300 randomly sampled chunks, a Qwen3-4B model generates 284 retained synthetic question-answer pairs using Environmental, Social, and Governance personas. Those pairs represent 181 distinct reports.

The reported comparison evaluates the first 100 pairs against all seven generators, producing 700 model-query evaluations. These are repeated evaluations of the same 100 questions, not 700 independent questions.

Each generator uses a common pipeline:

1. Qwen3 embeddings represent document chunks and queries.
2. Cosine similarity selects the single highest-ranked chunk.
3. The generator receives that chunk with an identical grounding prompt.
4. RAGAS scores the generated response using a Qwen3-based judge.

This configuration helps isolate generator-level differences, although decoding defaults are not fully standardized.

It also places a structural limit on the information available to the generator.

The retrieved context contains only one 1,000-character chunk, with 200-character overlap used during document preparation. A question requiring evidence from several disclosures may not be answerable from that single passage, regardless of the generator’s reasoning capabilities.

The retrieval metrics reinforce the need to examine evidence coverage. Context relevance ranges from 0.965 to 0.985, yet context recall is only 0.578–0.605.

These scores measure different properties, so their numerical magnitudes should not be treated as directly interchangeable. Still, the combination indicates that highly relevant retrieved material need not recover all the information represented in the reference answers.

The paper does not test larger retrieval windows, hybrid search, reranking, or alternative chunk sizes. Those approaches are reasonable candidates for subsequent evaluation, not demonstrated remedies.

Furthermore, image-only tables and figures are excluded by the text-extraction pipeline. For ESG workflows involving quantitative disclosures, this creates an additional document-coverage boundary before generation begins.

The evaluation resource is valuable, but its judge is not independent

Beyond model comparison, the study provides a reusable evaluation resource: processed ESG reports, synthetic questions, reference answers, evaluation outputs, and implementation scripts.

The authors release these materials through their ESG-RAG benchmark repository. This makes the study a potential starting point for organizations developing domain-specific evaluation procedures.

However, the evaluation design introduces an important validity concern.

Qwen3-4B participates in three roles: generating synthetic question-answer pairs, serving as one of the seven evaluated models, and judging responses through RAGAS.

This overlap creates a possible self-enhancement bias. A reference generator and evaluator from the same model family may favor certain response patterns. The study acknowledges the concern but does not measure its magnitude using independent judges.

Consequently, Qwen3’s leading factual-correctness score is an observed result under the reported protocol, not proof of unbiased superiority.

Other boundaries also affect deployment interpretation. The evaluation uses synthetic rather than authentic professional questions, takes the first 100 generated pairs without independent stratification, and reports neither human validation nor repeated-run uncertainty. It also lacks a direct non-RAG baseline, so it cannot establish how much retrieval improves accuracy over answering without retrieved documents.

The results are therefore moderately informative about the tested generators within this configuration, but insufficient to establish general rankings or production reliability.

Model selection should follow the reporting decision

For organizations implementing ESG assistants, the useful next step is a task-specific validation process rather than immediately adopting the highest-scoring checkpoint.

For disclosure navigation, where analysts search reports and inspect relevant passages themselves, answer relevancy and source grounding may carry substantial weight. A model’s ability to identify and explain pertinent disclosures can improve information access without necessarily authorizing its output as a verified reporting fact.

For structured KPI extraction, the acceptance criteria should be stricter. The system needs verified values, units, reporting periods, definitions, and source locations. A response that is relevant but disagrees with the validated reference should fail the extraction test.

For consequential reporting or compliance workflows, generated statements should remain subject to evidence checks and appropriate human approval. This is a proposed governance response to the study’s limitations, not an intervention experimentally evaluated by the authors.

A credible internal evaluation would therefore combine representative real questions, independently verified reference answers, retrieval-coverage tests, and generator comparisons. Model versions, decoding settings, judge identities, and retrieval configurations should be recorded so that results remain auditable.

Compact models such as Gemma4-e2b also deserve consideration during feasibility testing. Its competitive scores demonstrate that larger nominal parameter counts do not guarantee better results here. However, the paper does not provide sufficiently comprehensive per-model latency, energy, or serving-cost comparisons to establish an economic advantage.

The procurement decision should be informed by measured performance under the organization’s actual document distribution and operating constraints.

Better ESG answers require better evidence tests

This benchmark contributes two things: a controlled comparison showing that different models lead different answer-quality dimensions, and a reusable resource for extending ESG-specific evaluation.

Its strongest finding is not the identity of any winning model. It is the persistent gap between relevance-oriented scores and reference-answer factual correctness under the evaluated conditions.

For operational ESG systems, selecting a generator is only part of the validation problem. Retrieval must expose sufficient evidence, evaluation must distinguish grounding from factual agreement, and consequential outputs must be checked against independently established facts.

A relevant answer is a useful starting point. Whether it is sufficiently complete and correct for the intended reporting decision requires a separate test.

Cognaptus: Automate the Present, Incubate the Future.