FAITH in Numbers: Stress-Testing LLMs Against Financial Hallucinations

TL;DR for operators

FAITH is useful because it changes the hallucination question from “Does the model sound right?” to “Can the model reconstruct a known financial number from the exact tables and surrounding text that justify it?”¹ That sounds modest. It is not. In finance, modest is usually where the damage hides.

The paper’s core move is a masked-span test. Take a real annual report, mask a numeric value in the management discussion, provide the relevant tables and surrounding sentences, and ask an LLM to fill in the blank. If the answer is inside the document or derivable from it, any wrong answer is not a failure of web search, retrieval freshness, or missing world knowledge. It is an intrinsic hallucination: the model contradicts the supplied context.

The authors build FAITH around three filters: the masked value must be unique, consistent with the source document, and answerable from the provided context. They then evaluate models with a precision-relaxed matcher that recognises equivalent numeric formats and checks units. This matters because $1.23 billion and $1,230 million should not be treated as different answers, while $150 and $150 million absolutely should. Apparently, finance remains picky about orders of magnitude. A nuisance, I know.

The main benchmark uses 2024 S&P 500 10-K filings, focusing on Item 7, Management’s Discussion and Analysis. After processing and filtering, the main split covers 453 companies and 2,406 answerable spans. The model results show a clear hierarchy: Claude-Sonnet-4 reaches 95.6% overall accuracy on the main split, Gemini-2.5-Pro reaches 91.9%, GPT-4.1 reaches 89.2%, and many smaller open models fall far lower. But the practical lesson is not “buy the top model and go home.” Even high-performing models retain non-trivial errors, especially when the answer requires multi-step financial reasoning rather than direct lookup.

For business teams, the most valuable part of FAITH is not the public leaderboard. It is the recipe. A bank, asset manager, insurer, auditor, or corporate finance team could adapt the same mechanism to its own internal reports: mask known numbers, filter for answerability, test models under controlled conditions, classify reasoning difficulty, and track failure modes before letting generative AI near production workflows. This is model risk management with fewer vibes. Progress already.

The problem is not missing information; it is disobeying the information already given

Many discussions of hallucination in financial AI drift toward retrieval. The model lacked the right filing. The index was stale. The vector search found the wrong passage. The source did not include the latest quarter. These are real problems, but FAITH targets a more uncomfortable one: what happens when the right information is already there?

That distinction matters. If a model invents a fact because it never received the relevant document, the remedy is partly an information supply problem. Improve retrieval. Add citations. Refresh the corpus. Fix the data pipeline. But if the model receives the correct table and still reports the wrong number, the failure is deeper. The model has failed to remain faithful to its input.

In the paper’s terminology, this is an intrinsic hallucination. The output contradicts or misrepresents the provided context. In financial workflows, intrinsic errors are particularly dangerous because they look operationally clean. The model can quote the right company, discuss the right fiscal year, and use the right financial vocabulary, while quietly dropping a scale term, skipping a debt adjustment, or substituting a superficially plausible calculation.

FAITH is designed to catch exactly that class of failure. It does not ask whether an LLM knows finance in the abstract. It asks whether the model can read a specific financial context, recover a specific missing number, and preserve the numerical and unit structure that makes the answer meaningful.

This is a better stress test for production finance than generic factuality benchmarks, because most real deployments are not trivia contests. They are context-bound tasks: summarising a report, checking a covenant, extracting a metric, reconciling a table, drafting a commentary, or supporting an analyst who is trying not to turn one decimal place into a board-level incident.

FAITH works by masking known numbers, not by inventing questions

The benchmark design is elegant because it avoids a common weakness in evaluation datasets: hand-crafted questions that may not resemble the way financial information actually appears in documents.

FAITH starts with real financial reports. The authors use 10-K annual reports from S&P 500 companies filed in 2024, extract Item 7, Management’s Discussion and Analysis, and process the section into tables, table pre-texts, surrounding sentences, and candidate numeric spans. The model receives a sentence with one numeric span replaced by [MASK], plus the surrounding financial context. Its job is to reconstruct the original value.

That makes the benchmark close to a realistic analyst workflow. The model is not solving a synthetic word problem about a fictional company selling widgets. It is working inside the messier structure of corporate reporting: tables, explanatory text, units, periods, percentages, debt figures, ownership stakes, and enough formatting quirks to remind everyone that financial filings were not designed for the emotional comfort of machine learning systems.

The mechanism has four main layers:

The paper’s most important design choice is the filtering. A masked value must satisfy three assumptions.

First, it must be unique. If the sentence says “the company’s [MASK] improved,” there may be several plausible completions. Revenue, margin, operating income, liquidity, and management’s optimism all have a way of “improving” when someone writes creatively enough. FAITH excludes that kind of ambiguity.

Second, the masked value must be consistent with the document. If the sentence itself contains a mistaken number that contradicts the table, using that number as ground truth would punish a model for being more faithful than the report. That would be a splendidly bureaucratic failure mode, but not a useful benchmark.

Third, the value must be answerable from the provided context. If the missing value requires information outside the tables and surrounding sentences, the model should not be expected to derive it.

These assumptions sound like housekeeping. They are not. They determine whether the benchmark is measuring hallucination or merely measuring ambiguity. In finance, those are different animals, though both bite.

The pilot study validates the filter, not the whole deployment story

Before scaling the dataset, the authors run a pilot study to test whether LLMs can help annotate answerability. This is best understood as a validation step for dataset construction, not as the main evidence about model reliability.

The pilot uses 1,124 text spans from the Item 7 sections of 10-K reports from nine companies across different industries. At least two financial experts label each span for answerability, with a senior reviewer adjudicating disagreements. The human labels show high agreement, with Fleiss’ kappa of 0.905. The result is 300 answerable spans and 824 unanswerable spans.

The authors then ask three frontier models—GPT-4.1, Claude-Sonnet-4, and Gemini-2.5-Pro—to annotate answerability. When all three agree, consensus is highly reliable. For unanswerable spans, unanimous “No” agreement correctly identifies 626 out of 627 cases. For answerable spans, unanimous “Yes” agreement correctly identifies 276 out of 287 cases.

This supports the paper’s scaling decision: use unanimous agreement among strong LLMs to filter answerable spans for the larger dataset. It does not prove that LLMs are generally reliable financial annotators under all conditions. The consensus rule is deliberately conservative. It keeps cases where the models agree and discards disagreement cases.

That choice makes the benchmark cleaner, but it also creates a boundary. The most ambiguous or difficult spans may be excluded precisely because the annotators disagree. The authors acknowledge this limitation. From a business perspective, the implication is simple: FAITH-style tests are excellent for measuring failures on clean, answerable tasks. They may understate risk in messy internal documents where ambiguity, incomplete tables, and inconsistent reporting formats are part of the daily entertainment package.

The matcher is where financial realism enters the benchmark

A naive benchmark would compare the model’s output string to the original masked string. That would be quick, simple, and wrong often enough to be annoying.

Financial numbers have equivalent surface forms. $1,230 million, $1.23 billion, and USD 1.23bn may represent the same value. A string matcher would mark these as different. FAITH instead uses a precision-relaxed matching process with unit groups.

The numeric matcher normalises values into a base representation, interprets scale indicators such as million or billion, and compares values after rounding according to the coarser precision of the prediction and ground truth. The unit matcher checks whether the relevant non-scale units match, using alias groups such as dollar symbols, USD, dollars, or similar unit expressions.

This is not a decorative technical detail. It determines whether the reported accuracy reflects actual financial correctness. A model should not lose credit for a harmless formatting difference. It should lose credit for changing millions into raw dollars, percentages into currency, or per-share figures into aggregate amounts.

The case study later in the paper shows why this matters. One recurring error is scale failure: the model identifies a plausible number but drops the magnitude. In Llama-3.3-70B, correcting just this error type would raise value accuracy from 37.0% to 57.7%. That is not a rounding issue. That is a model grounding issue wearing a formatting costume.

For operators, scale errors are especially treacherous because they often survive casual review. A number may look familiar, appear in the right sentence, and still be off by a factor of one million. This is why “the model cited the table” is not enough. It must preserve the table’s arithmetic meaning.

Four reasoning classes expose where the model stops being careful

FAITH divides tasks into four reasoning scenarios. This is where the benchmark becomes more useful than a single accuracy number.

The categories are not merely labels for presentation. They are a diagnostic instrument. If a model performs well on direct lookup but collapses on multivariate calculation, the team learns that the issue is not basic extraction. It is chained reasoning across table and text.

The paper uses human scenario labels for the pilot split. For the main split, it takes a more unusual approach: models generate step-by-step rationales and classify their own reasoning scenario; the final scenario label for a span is aggregated from models that correctly predict the answer. This is a practical solution, but it deserves careful interpretation. The labels are grounded in successful model reasoning, not independent human annotation. That is reasonable for scaling, but it means the scenario breakdown is partly downstream of model behaviour.

Still, the broad pattern is clear. Accuracy falls as reasoning complexity rises. Direct lookup is comparatively manageable. Multivariate calculation is where many models reveal that they were not so much “reasoning” as politely rearranging nearby numbers.

The main results are a hierarchy, but not the kind procurement teams want

The main split contains 2,406 answerable spans from 453 companies. The average context length is over 12,800 characters, and each document context contains an average of 19.2 tables. This is not a toy table benchmark. It is closer to the kind of dense reporting environment where financial AI systems are actually expected to behave.

The model ranking shows a clear top tier on overall accuracy:

Two readings are tempting. The lazy reading is that frontier proprietary models are “good enough.” The more useful reading is that the acceptable model depends on the task’s tolerance for error and the type of reasoning required.

Claude-Sonnet-4 reaches 95.6% overall on the main split. That is strong. It also means roughly one in twenty-five benchmark tasks fails. In a low-stakes summarisation assistant, that may be acceptable with review. In regulatory reporting, covenant monitoring, risk disclosure, credit memo generation, or investment committee materials, the tolerance may be much lower.

Gemini-2.5-Pro has lower overall accuracy than Claude-Sonnet-4 but performs very strongly on the small multivariate slice, reaching 90.0%. GPT-4.1 performs well overall but drops to 30.0% on multivariate examples. That contrast is operationally important. A procurement scorecard based only on overall accuracy could select a model that looks strong across the full dataset but fails in exactly the kind of multi-step reasoning that senior analysts wanted to automate.

There is also a sample-size boundary. The main split includes 1,606 direct lookup cases, 635 comparative cases, 135 bivariate cases, and only 10 multivariate cases. The multivariate results are therefore useful as a warning signal, not as a precise ranking of model capability. A 90.0% score on ten cases means one failure. An 80.0% score means two failures. The exact ordering may move with a larger sample. The existence of failures on these cases is the point.

The case study shows why “nearby arithmetic” is not financial reasoning

The paper’s qualitative case study is not the main evidence. It is a diagnostic illustration. Its purpose is to show what model failure looks like when aggregate accuracy numbers are no longer enough.

The target is a $20.2 million equity investment figure related to Fund V acquiring a 90% interest in an unconsolidated venture that purchased Mohawk Commons for $62.1 million. The table provides relevant details: Mohawk Commons has 18.1% ownership and $7.2 million of pro-rata mortgage debt.

Gemini-2.5-Pro reconstructs the required logic. It identifies the relevant table entry, infers total mortgage debt by dividing the pro-rata debt by the ownership percentage, subtracts that debt from the purchase price to derive total equity, and then applies the 90% fund interest to reach $20.2 million.

GPT-4.1 and Claude-Sonnet-4 do not. They rely on the sentence alone and calculate 90% of the $62.1 million purchase price, skipping the debt adjustment. The resulting answer is arithmetically plausible and financially incomplete. A very modern failure: correct multiplication, wrong concept.

This matters because many financial values are not directly printed as single table cells. They are latent variables implied by a combination of text, ownership percentages, debt allocations, acquisitions, and accounting structure. A model that grabs the nearest numbers and performs the most obvious operation may look competent until the workflow requires actual financial interpretation.

The lesson is not that Gemini-2.5-Pro is universally superior. It is that evaluation must include cases where the obvious calculation is a trap. Finance is full of those. They are called “details,” usually by the person who catches them after everyone else has presented the deck.

What FAITH directly shows, and what business teams should infer

The paper directly shows three things.

First, a masked-span framework can generate a scalable benchmark for intrinsic hallucinations in financial tables. The key is that the ground truth already exists in the source document. That makes evaluation less dependent on subjective judgement than open-ended financial QA.

Second, model reliability is stratified. The strongest proprietary systems perform much better than smaller open models on this benchmark. This is not shocking, but the numbers make the gap operationally visible.

Third, reasoning complexity drives failure. Direct lookup is easier. Multi-step financial reasoning remains fragile, and some models collapse on the hardest category.

Cognaptus would infer a fourth point for business use: FAITH is most valuable as a pattern for internal testing. Public benchmarks are useful for orientation, but they rarely match an institution’s actual documents, table styles, risk appetite, jurisdictional accounting conventions, or workflow constraints.

A practical FAITH-inspired internal evaluation would look like this:

This is not glamorous. It is also the part that may save money. Most AI failures in enterprise settings are not solved by one grand architectural revelation. They are reduced by dull, disciplined testing that catches predictable failure modes before they scale. Naturally, that is less fun to sell in a keynote.

The appendix prompts reveal a useful operational discipline

The appendices are implementation details, not a second thesis. They show how the authors prompt models for answerability annotation and financial metric prediction.

The answerability prompt asks the model to determine whether highlighted spans are answerable from the tables and sentence context, with concise reasoning. The prediction prompt casts the model as a financial analyst, asks it to reason step by step, identify formulas or table cells used, classify the reasoning scenario, and return a structured JSON answer.

For production teams, the important point is not to copy the prompts verbatim. The useful discipline is structured output plus explicit reasoning classification. Asking the model to provide the answer alone gives a number. Asking for the scenario, necessary metrics, and references gives evaluators more ways to identify why the number failed.

That said, teams should not confuse prompted reasoning with guaranteed faithful reasoning. Chain-of-thought-like explanations can be useful for debugging, but they are not proof that the model actually used the cited table correctly. FAITH treats the final value and unit as the measurable answer. The rationale is diagnostic support, not the ground truth.

Boundaries: the benchmark is strong, but not universal

FAITH is well designed for what it studies. Its boundaries are also clear.

The dataset is based on public U.S. company filings. That gives consistency and comparability, but it does not automatically cover private company reports, non-U.S. disclosures, bank-specific regulatory templates, insurer filings, management accounts, investor memos, or messy spreadsheet exports from the department no one wants to name.

The benchmark focuses on curated financial tables and surrounding text from MD&A sections. That is a sensible starting point, but many enterprise workflows involve less tidy document structures: scanned PDFs, slide decks, email attachments, versioned spreadsheets, incomplete footnotes, and tables whose headers were apparently designed during a fire drill.

The answerability filter uses unanimous agreement among frontier models. The pilot study supports this as a reliable scaling method, but it may exclude difficult cases where models disagree. In other words, the benchmark may be measuring clean answerable tasks more than the truly ambiguous edge cases that make production systems sweat.

The multivariate slice is small. The results strongly suggest that complex reasoning is risky, but exact model comparisons in that category should not be over-read. Ten main-split examples are enough to raise eyebrows, not enough to write procurement law.

Finally, FAITH evaluates intrinsic hallucination under a specific masked-span setup. It does not measure every risk that matters in financial AI: strategic misuse, overconfident narrative generation, regulatory interpretation, adversarial prompting, tool failures, retrieval misses, or human overreliance. A responsible model risk programme would treat FAITH-style tests as one layer, not the entire control environment.

The business value is not trust; it is calibrated mistrust

The word “faith” in FAITH is doing some heavy lifting. The framework is not asking organisations to trust LLMs. It is giving them a way to distrust models productively.

That is the right posture for financial AI. The goal is not to prove that a model is intelligent in general. The goal is to know which tasks it can perform, which tasks it cannot, which errors it makes repeatedly, and which workflows require human review, tool support, or outright prohibition.

The paper’s results are therefore both encouraging and inconvenient. Encouraging, because the best models are genuinely strong on many financial table tasks. Inconvenient, because the same models still fail on cases that look exactly like the cases enterprises most want to automate: multi-step, context-heavy, numerically precise reasoning across tables and prose.

The operational conclusion is plain. Do not evaluate financial LLMs only with generic benchmarks, vendor demos, or one-off analyst impressions. Build masked-span tests from your own documents. Track value accuracy, unit accuracy, scale errors, and reasoning class. Separate lookup from calculation. Separate calculation from financial interpretation. Then deploy accordingly.

The future of financial AI will not belong to the team with the most enthusiastic chatbot. It will belong to the team that knows, with boring precision, where the chatbot breaks.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast