Explaining the Explainers: Why Faithful XAI for LLMs Finally Needs a Benchmark

Hiring.

A candidate writes a personal statement. A screening model gives a score. A manager asks the AI system why. The explanation says work experience mattered most, education came next, and demographic variables barely moved the decision.

Everyone relaxes, because the explanation sounds reasonable.

That is the dangerous part.

A reasonable explanation is not necessarily a faithful explanation. A counterfactual edit that looks plausible is not necessarily a causal counterfactual. And a model that appears insensitive to demographic concepts may not be “fair”; it may simply have learned, or been aligned, to suppress visible sensitivity in the narrow setting being tested.

The paper behind this article, “LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals,” tackles a problem that explainable AI has politely walked around for years: how do we evaluate explanations when we do not know the true explanation?¹

Most explainability benchmarks are forced to use proxies. They ask whether explanation methods agree with one another, whether highlighted tokens look sensible, whether humans find the explanation convincing, or whether a generated counterfactual resembles a human-written edit. These tests are not useless. They are just not the same as asking whether the explanation is faithful to the model’s actual behavior.

LIBERTy changes the object being benchmarked. It does not ask, “Can we produce a nice explanation?” It asks, “When we know the data-generating mechanism, can an explanation method recover the causal effect of a concept on a model prediction?”

That small change is the whole paper. Everything else follows from it.

The benchmark starts by making the hidden mechanism explicit

The authors build LIBERTy around a structural causal model, or SCM. In plain terms, an SCM is a map of how variables generate one another. Some variables are concepts, such as gender, race, age, work experience, symptoms, or department. One variable is the label the model is trained to predict. Another variable is the generated text. The model being explained then reads that text and produces a prediction.

This matters because concept-based explanations are supposed to speak in human-meaningful units. They do not say “token 143 contributed 0.08.” They say “education influenced the hiring score,” or “headache influenced the disease prediction,” or “department influenced workplace violence risk.” That is exactly the language business users want. It is also exactly the language where unfaithful explanations can become dangerously persuasive.

LIBERTy’s move is to define the causal graph first, then generate text from it. The paper creates three datasets:

The graphs are not toy four-variable diagrams. Each has at least eight concepts, includes confounding and mediation, and supports longer causal paths. That is important because real business explanations rarely involve isolated features. “Age” may affect “work experience,” which may affect “quality,” which affects the model’s prediction. If an explanation method treats those relationships as flat independent variables, it may sound clean while missing the mechanism.

The authors then use GPT-4o as the text generator inside the SCM. Gemini is used earlier to help create templates and personas. The generator receives concept values, persona information, and a text template, then produces the natural language example using deterministic decoding. The deterministic part is not a boring implementation detail. It is the reason the counterfactual can be structural.

With stochastic decoding, the factual text and counterfactual text might differ because of uncontrolled generation noise. The story changes, the phrasing changes, the hidden narrative changes. Cute, but causally messy. LIBERTy instead keeps exogenous elements fixed: same persona, same template, same decoding process. Then it intervenes on a concept and propagates the change through the causal model before regenerating the text.

So the benchmark is not merely editing a sentence. It is running Pearl’s counterfactual logic in a synthetic text world: fix the background conditions, change the concept, propagate the consequences, observe the new model prediction.

That gives the paper its core evaluation target: a reference causal concept effect.

A plausible edit is not the same thing as a causal counterfactual

Here is the misconception LIBERTy quietly attacks: if a counterfactual text looks plausible, it must be a good explanation.

Suppose a CV says a candidate has a bachelor’s degree and five years of experience. A counterfactual generator is asked to change education to doctorate. A human-like editor may change “bachelor’s degree” to “doctorate” and keep everything else fixed. That is a minimal edit. It looks controlled. It feels scientific.

But if the underlying causal model says education also affects work experience, volunteering, certification, or candidate quality, then keeping everything else fixed may be the wrong counterfactual for total causal effect. It answers a different question: “What if this word changed and nothing causally downstream changed?” That is not necessarily the question a decision-maker meant to ask.

LIBERTy makes this difference visible. It distinguishes between an edit that resembles what a human annotator might write and a counterfactual generated by an intervention in the data-generating process.

This explains one of the paper’s most interesting findings. Earlier work found that LLM-generated counterfactuals could perform strongly when compared against human-written counterfactuals. LIBERTy reverses the ranking. On this benchmark, LLM counterfactual generation performs worse than matching-based methods.

That is not because LLMs suddenly forgot how to write. The problem is more embarrassing: they are good at writing plausible edits, but the benchmark rewards tracking the DGP. When the reference is a human edit, the LLM can mimic the human editing style. When the reference is a structural intervention, mimicry loses its crown. Tiny violin, very causal.

LIBERTy evaluates explanations by effect size and by ordering

The paper uses two main evaluation ideas.

The first is error distance. For a local explanation, the benchmark compares the explanation method’s score for a concept change with the reference individual causal concept effect, or ICaCE. Lower error distance is better.

The second is order-faithfulness, introduced by the paper. This is more directly useful for many real explanation settings. In practice, managers and auditors often care less about the exact numerical causal effect and more about whether the explanation ranks drivers correctly. Did the method identify that work experience mattered more than race? Did it rank light sensitivity above fever? Did it preserve the relative importance of concept changes?

Order-faithfulness asks whether the explanation preserves the sign and relative ordering of causal effects. Higher is better, with 1 as perfect.

This is a useful metric because many explanation methods output scores on arbitrary scales. One method’s “0.7” is not another method’s “0.7.” But ranking matters. If an explanation method cannot reliably say which concepts mattered more, its business usefulness is limited, no matter how polished the dashboard looks.

The main experiment is a benchmark, not a beauty contest

The authors evaluate five predictive models:

They then test eight explanation methods across four families:

The main result is not subtle: fine-tuned matching performs best overall.

For local explanations, FT Match achieves the best average results: 0.34 error distance and 0.74 order-faithfulness. Counterfactual generation averages 0.55 error distance and 0.49 order-faithfulness. LEACE performs worse in the reported local setting, with 0.65 error distance and 0.46 order-faithfulness, though the paper applies it only where erasure is well-defined.

For global explanations, the same pattern appears. Mean global order-faithfulness is highest for FT Match at 0.85, followed by ConVecs at 0.82, Approx at 0.77, ST Match at 0.73, PT Match at 0.69, LEACE at 0.62, counterfactual generation at 0.61, and ConceptSHAP at 0.41.

The interpretation is not “matching is magic.” The better reading is that matching methods search within candidate texts sampled from the same benchmark distribution. FT Match uses embeddings from a model fine-tuned on the task, which likely gives it neighborhoods that better reflect the relevant causal and predictive structure. The method wins because it stays closer to the DGP than a free-form LLM editor does.

That is a useful lesson for businesses. In explainability, bigger language models do not automatically beat smaller, narrower, distribution-aware tools. Sometimes the humble method that knows the local data-generating regime beats the eloquent model that knows the internet.

Annoying for demo videos. Helpful for governance.

The appendix tests explain why “just prompt the LLM better” is not enough

One appendix experiment deserves attention because it prevents a lazy objection.

The authors test four prompting strategies for counterfactual generation on CV screening:

This is best read as an implementation and diagnostic comparison, not the paper’s main thesis. Its purpose is to see whether more causally informed prompting improves LLM-generated counterfactuals.

The answer is: somewhat, but not enough to change the big picture. The most causally informed prompt is selected for the main experiments, but counterfactual generation still trails matching methods in the headline results.

This matters because “add causal instructions to the prompt” is the default reflex of the current AI industry. LIBERTy suggests that the issue is deeper. If the LLM generating the counterfactual is not actually operating inside the target DGP, it may still produce edits based on heuristic assumptions. The prompt can describe confounders and mediators, but the model must infer how they should change. That inference can be plausible and still wrong relative to the benchmark’s structural mechanism.

Prompting helps the actor understand the script. It does not make the actor the playwright.

Human validation checks realism, not causal truth

The paper also conducts human validation of generated examples. Annotators assess whether texts include the intended concept values, whether they are coherent and fluent, whether they are relevant and logically consistent, and whether counterfactual pairs feel like plausible variants from the same person in a parallel scenario.

The results are strong: average concept agreement is 94.2%, coherence is 4.79/5, fluency is 4.85/5, relevance is 4.77/5, consistency is 4.92/5, and counterfactual plausibility is 4.44/5.

This is useful evidence, but it should be interpreted correctly. Human validation here is mainly a data quality and realism check. It supports the claim that the synthetic texts are usable and not obviously broken. It does not prove that the SCM mirrors reality, and the authors do not claim that it does.

This distinction is important for business readers. A synthetic benchmark can be realistic enough to test explanation methods without being a model of the real world. In fact, its value comes from being controlled. The point is not to discover the true causal structure of hiring, medicine, or workplace violence. The point is to create a world where the causal structure is known, then ask whether explanation methods can recover it.

That is benchmark logic. Not sociology. Not clinical science. Not employment law. Please do not submit the causal graph to HR as policy.

The sensitivity analysis is about model behavior, not explanation quality

LIBERTy can do more than score explanation methods. Because it has structural counterfactuals, it can also measure how sensitive each model is to concept interventions.

This is a different experiment from the explanation benchmark. It asks: when a concept changes, how much does the model prediction move?

The paper reports that zero-shot LLMs, including Llama-3.1-8B and GPT-4o, show lower sensitivity to concept changes, especially demographic concepts such as race, gender, and age. The authors suggest this may reflect post-training alignment or mitigation behavior. Fine-tuned models tend to track the data’s effects more closely, with Qwen2.5-1.5B standing out among the fine-tuned models in their analysis. Still, none perfectly matches the gold effects.

For example, in workplace violence prediction, the true sensitivity effects for selected demographic concepts are reported as race 0.484, gender 1.271, and age 1.154. Qwen2.5 has corresponding scores of 0.691, 1.314, and 1.045, closer than several alternatives. Llama-3.1 is much flatter on the same concepts: 0.224, 0.227, and 0.226.

This result has a tempting but dangerous business reading: “Great, aligned LLMs are less sensitive to demographics.”

Careful.

Low measured sensitivity in this benchmark does not prove fairness. It means the model’s prediction changed less under these structural interventions in this synthetic setting. That may reflect mitigation. It may also hide other routes by which sensitive information still enters decisions. It may reduce explicit demographic responsiveness while leaving proxy effects untouched. It may be exactly what a compliance officer wants in one context and exactly what a causal auditor needs to investigate in another.

The business lesson is not “use aligned LLMs and sleep well.” The business lesson is “sensitivity analysis must be separated from explanation evaluation.” One asks how the model reacts. The other asks whether an explanation method accurately describes that reaction. Mixing the two is how governance slides become decorative wallpaper.

What LIBERTy directly shows, and what businesses should infer

The paper directly shows three things.

First, structural counterfactuals can be generated at scale by defining an SCM, sampling concepts, grounding text generation with personas and templates, and intervening on concepts while holding exogenous context fixed.

Second, when concept-based explanation methods are evaluated against these structural references, fine-tuned matching methods outperform the tested LLM counterfactual generation methods, both locally and globally.

Third, there is still large headroom. Even the best local methods land around 0.3 error distance and 0.7 order-faithfulness, where perfect would be 0 and 1, respectively. Global rankings improve, but every method still misses at least one gold top concept in the selected examples reported by the paper.

From those results, Cognaptus would infer a practical audit pathway for firms using LLMs in high-stakes text workflows:

The ROI relevance is not that LIBERTy gives companies an instant compliance button. It does not. The value is cheaper diagnosis of explanation methods before they are placed in front of managers, doctors, HR reviewers, loan officers, or auditors.

A bad explanation in a low-stakes chatbot is irritating. A bad explanation in a screening or risk workflow can create false confidence. False confidence is expensive. It leads to bad escalation rules, bad audit trails, bad vendor evaluations, and the inevitable meeting where everyone suddenly remembers that “explainable” was never defined.

The business boundary is synthetic, concept-based, and model-relative

LIBERTy’s limitations are not footnotes to be sprinkled everywhere. They are central to how the benchmark should be used.

First, the text is synthetic. The authors validate quality, but synthetic text is still generated text. It may not capture how real applicants, patients, employees, or customers express information. The paper argues that this limitation is becoming less restrictive as more real-world data becomes LLM-generated. That may be true in many workflows, but it is not universal.

Second, the causal graphs approximate reality. The authors are explicit that the SCMs are not meant to recover the true causal structure of disease, hiring, or workplace violence. They are plausible, literature-informed structures designed for controlled evaluation. That is enough for benchmarking explanation methods. It is not enough for making real-world causal claims.

Third, the benchmark focuses on concept-based explanations. It does not evaluate all explainability forms. Token attribution, free-text rationales, chain-of-thought-style justifications, mechanistic interpretability, and user-facing explanation UX are separate questions.

Fourth, the benchmark is model-relative. The causal effects being measured are effects on the explained model’s prediction. An explanation can be faithful to a biased model. Faithfulness is not fairness. Faithfulness means the map matches the machine. It does not mean the machine deserves approval.

That last sentence should probably be printed on procurement documents.

Why this paper matters more than another leaderboard

The easiest way to summarize LIBERTy is to say it introduces a new benchmark and reports that FT Match wins. That summary is accurate and mostly unhelpful.

The deeper contribution is methodological. LIBERTy shifts explainability evaluation from plausibility to mechanism. It says: if you want to know whether an explanation is faithful, you need a reference causal effect. If reality will not give you one, build a controlled data-generating process where the reference is known. Then test whether explanation methods recover it.

This is especially relevant for LLM-era systems because natural language makes everything look more convincing. A generated explanation can sound balanced, specific, and professional while being causally unmoored. A generated counterfactual can look like a careful edit while violating the intervention it claims to represent. A model can appear demographically restrained while remaining opaque in its proxy behavior.

LIBERTy does not solve those problems. It gives researchers and governance teams a sharper instrument for finding them.

For businesses, the practical message is simple: do not buy explanations by the paragraph. Buy them by the test.

An explanation method should not merely produce a sentence that a stakeholder can understand. It should survive controlled counterfactual evaluation. It should rank concept influence correctly. It should distinguish local behavior from global importance. It should reveal whether the model is sensitive to concepts, and separately whether the explanation captures that sensitivity.

That is less glamorous than a chatbot saying, “Here is why I made this decision.” It is also much closer to what accountable AI systems actually need.

The future of explainable AI is not more eloquent explainers. It is explanations with receipts.

Cognaptus: Automate the Present, Incubate the Future.