Numbers Don’t Speak for Themselves: How LLMs Interpret the Soul of Financial Reports

TL;DR for operators

Financial-report analysis is one of those jobs where the output can sound competent long before it is useful. A model can summarise a 10-K fluently, mention strategy, risk, customers, and competitive position, and still fail the only test that matters: can a finance team rely on it repeatedly, under pressure, across filings?

A new pilot study on LLMs and 10-K Business sections gives a useful answer: do not select a financial-analysis model using one score.¹ The study compares GPT-4, Claude 4 Opus, Gemini Pro, Perplexity, and DeepSeek-V2 on open-ended questions over Item 1 of annual 10-K filings from Apple, Microsoft, Amazon, Alphabet, Nvidia, Meta, and Tesla. It evaluates the models through human ratings, lexical similarity, semantic similarity, and behavioural diagnostics.

The result is not a clean horse race. GPT has the highest average human evaluation score at 4.08. Gemini dominates lexical-overlap measures such as ROUGE and Jaccard. Semantic similarity is more ambiguous: the paper names Claude as best in one summary table, while another aggregate table gives Perplexity the highest cosine similarity. Behavioural diagnostics show that model agreement changes across companies and years, with Microsoft disclosures producing relatively high agreement and Amazon 2024 producing the lowest reported mean agreement.

For banks, asset managers, auditors, investor-relations teams, and compliance groups, the operational message is simple but inconvenient: model procurement should look more like validation than shopping. Test models on your own disclosure tasks. Use more than one metric. Include human review. Track model-version drift. And never let a polished answer masquerade as reliability. Finance has suffered enough from confident prose already.

The leaderboard is the least interesting part

The obvious reading of this paper is: “Which model won?”

That is also the least useful reading.

The study does report apparent winners. GPT receives the highest average human score. Gemini produces the strongest lexical overlap with reference answers. Perplexity and Claude look stronger under semantic alignment than under surface-word matching. DeepSeek is concise but weaker on relevance and factual accuracy. If this were a procurement slide, someone would be tempted to put five logos in a grid and crown a champion.

That temptation should be resisted with unusual force.

The paper’s value is not that it tells us which model is “best at finance”. It shows that financial LLM performance changes depending on what you mean by performance. Human reviewers reward one bundle of traits: relevance, completeness, clarity, conciseness, and factual accuracy. ROUGE and Jaccard reward token overlap with reference answers. Sentence-BERT cosine similarity rewards semantic closeness. Behavioural diagnostics reward consistency across prompts, companies, and time.

Those are not interchangeable tests. They are different instruments pointed at the same machine.

A model can look excellent because it uses similar wording to a reference answer. That does not prove it has the best financial judgement. A model can score well semantically while using different phrasing. That does not prove it is safer for audit workflows. A model can be concise. That is lovely, unless it is concisely omitting the point.

So the business question is not “Which model won?” It is “Which failure mode can your workflow tolerate?”

What the study actually tested

The paper evaluates five transformer-based LLMs:

The data consist of Item 1 Business sections from 10-K filings for seven large technology firms: Apple, Microsoft, Amazon, Alphabet, Nvidia, Meta, and Tesla. The study uses three years of filings, producing 21 documents. Nvidia’s fiscal-year timing is handled separately: its 2025 filing is treated as temporally aligned with the 2024 disclosures of the other firms.

Each document is paired with 10 interpretive questions. These are not merely “find the sentence” questions. They ask about strategic goals, competitive position, growth strategy, operational problems, business model clarity, stakeholder value proposition, over-reliance on business lines, and forward-looking orientation.

That design matters. The study is testing whether LLMs can interpret the narrative section of a filing, not whether they can extract a number from a table. Item 1 is where companies explain what they think they are doing. Sometimes they explain it clearly. Sometimes they perform the corporate equivalent of fog-machine theatre. The model’s job is to turn that prose into an answer without importing outside knowledge.

The paper uses one base prompt structure:

You are analysing the Business section of a firm’s 10-K filing. Use only the information contained in the passage. Do not introduce external knowledge. [10-K Business Section Excerpt] Question: [Q_i] Provide a concise, factual answer.

Each query is run in a fresh chat session to avoid context leakage. The study reports 210 responses per model, 1,050 model outputs in total, and 5,250 human ratings from five annotators.

The primary results use non-chain-of-thought responses. A separate diagnostic chain-of-thought mode is included to examine reasoning structure, verbosity, and stability, but those traces are not scored and are not shown to annotators. That distinction is important: the CoT component is an exploratory diagnostic extension, not the main evidence for model ranking.

The four lenses disagree, which is the whole point

The study’s comparison-based structure is useful because the findings only make sense when placed side by side. Here is the practical reading.

This is where the lazy summary collapses. If you ask “who won?”, the answer depends on which column you care about.

For a compliance team, the low inter-rater reliability may matter more than the average score. If humans cannot consistently agree on which answer is better, then the evaluation protocol needs stronger rubrics, domain-specialist review, or task-specific acceptance criteria.

For a retrieval-augmented investor-relations assistant, lexical overlap may be less valuable than factual grounding and source faithfulness. A model that paraphrases well may be more useful than one that copies reference wording.

For a longitudinal monitoring workflow, behavioural stability matters. If the same type of prompt produces materially different interpretations across years or companies, the system may need trend-level controls rather than answer-level checks.

In other words: the paper is not a menu of model rankings. It is a warning label for model evaluation.

Human judgement favours GPT, but the judges disagree

The paper’s human evaluation scores look reassuring at first glance. GPT has the highest average score at 4.08. Perplexity and Gemini both average 4.01. Claude follows closely at 3.99. DeepSeek averages 3.92.

On a five-point scale, that seems like a narrow spread. It is. The difference between “best” and “worst” in average human score is only 0.16 points.

The category-level story is more informative. GPT is described as strong on relevance, completeness, and clarity. Claude is strong on factual correctness but slightly weaker on conciseness and completeness. Perplexity is balanced, without dramatic highs or lows. DeepSeek is the most concise, but its relevance and factual accuracy are lower. Gemini is comprehensible and sometimes factually strong, but its low conciseness suggests too much verbal freight.

That already gives operators a better question than “best model?”:

But the inter-rater reliability results complicate the story. Krippendorff’s alpha and ICC values are low across all dimensions, often near zero and sometimes negative. The paper interprets this as evidence that expert judgements frequently differ and that between-rater variability dominates between-item variability.

That is not a minor statistical footnote. It changes the status of the human ranking.

The correct reading is not “GPT is objectively best”. It is closer to: under this prompt design, sample, and reviewer group, GPT produced answers that were marginally preferred on average, but the human evaluation itself is noisy.

That is exactly what real financial AI deployment looks like. Different reviewers value different things. One analyst rewards completeness. Another punishes verbosity. One compliance reviewer focuses on unsupported inference. Another asks whether the answer is useful enough for a meeting note. The model is not being judged against a Platonic filing oracle. It is being judged inside an organisational culture with inconsistent standards. Delightful, as always.

The business response is not to abandon human evaluation. It is to make it less vague. A useful internal rubric should separate factual grounding, source citation, analytical inference, omission risk, and actionability. “Good answer” is not a governance category.

Gemini wins the wording contest; that is not the same as winning finance

Automated metrics tell a different story from human ratings.

Gemini leads the lexical metrics. The paper reports Gemini as best on ROUGE-1 with an average score of 0.56, ROUGE-2 at 0.22, ROUGE-L at 0.16, and Jaccard around 0.21–0.22. These metrics reward overlap with reference responses: similar words, similar phrases, similar token sets.

That is useful, but it is a narrow kind of usefulness.

In financial-report analysis, lexical similarity can be desirable when the task is close to extraction or standardised reporting. If the target answer should preserve regulatory language, a model that stays close to reference phrasing may reduce interpretive drift. For example, investor-relations teams may prefer wording that remains close to the filing when producing internal briefing notes. Legal teams may also prefer less paraphrastic adventure. There are already enough ways for lawyers to be unhappy.

But lexical overlap can also reward the wrong behaviour. A model can sound aligned because it repeats the same terms while missing the strategic implication. Another model can use different wording while preserving the meaning. This is why the paper’s Figure 4, a joint distribution of cosine similarity, Jaccard similarity, and ROUGE-L, matters as a visual diagnostic rather than decorative chartwork. Its purpose is to show that lexical fidelity and semantic proximity occupy different spaces. Models that cluster on one axis do not necessarily cluster on another.

The operational lesson is direct: ROUGE is not a financial analyst. It can measure wording resemblance. It cannot decide whether an answer captures the economic substance of a business description.

Semantic similarity is more useful, but not clean enough to be a throne

Semantic metrics are meant to get past surface wording. The study uses Sentence-BERT cosine similarity to compare model outputs with reference answers. In principle, this should better capture whether two answers mean similar things even when they use different words.

The paper’s semantic results are not perfectly tidy. Table 5 reports Claude as the best model for cosine similarity, with an average score of 0.68 and a range up to 0.80. Table 6, however, reports average cosine similarity as Perplexity 0.71, GPT 0.68, Claude 0.67, Gemini 0.63, and DeepSeek 0.59.

The safest interpretation is not to force a single semantic winner. The safer interpretation is this: semantic-alignment metrics disrupt the lexical leaderboard. Gemini’s lexical strength does not translate into the strongest semantic positioning. GPT, Perplexity, and Claude appear more competitive on meaning than they do on word overlap.

This matters for business because many financial-report tasks are semantic rather than lexical. When a model is asked, “What is the company’s competitive position?” it must connect claims about products, markets, customers, dependencies, and strategic direction. It is not enough to recite filing vocabulary. The answer must preserve meaning across abstraction.

Still, semantic similarity is not magic. It depends on the reference answer. It depends on the embedding model. It may treat two answers as close even when one omits a crucial caveat. It may not understand the difference between “management says it plans to expand” and “the company is likely to expand”. In finance, that distinction is not pedantry. It is the border between analysis and hallucinated confidence.

So semantic scores should be part of the evaluation stack, not the evaluation stack.

Stability is where model selection starts looking like governance

The behavioural diagnostics are the most business-relevant part of the paper because they move beyond answer quality into repeatability.

The study examines pairwise cosine similarity between models and prompt-level variability across companies and years. This is not asking, “Was this answer right?” It is asking, “Do models interpret the same disclosure in similar ways, and does that similarity hold across disclosure contexts?”

The results show that cross-model agreement varies meaningfully.

For Microsoft, the reported mean cosine similarities are high, around 0.84–0.85 with low standard deviation. For Amazon 2024, mean similarity drops to 0.708 with a standard deviation of 0.078, the lowest and most variable case reported in the table. Pairwise model similarities also vary: GPT and Claude show high agreement in several company datasets, such as 0.84 for Apple and 0.87 for Microsoft, while some pairs involving Gemini or DeepSeek fall closer to 0.69–0.72 in certain cases.

This supports a practical inference: the filing matters. Models do not process “10-Ks” as one homogeneous genre. They respond to the structure, clarity, and content of a specific disclosure. When the source text is stable, clear, and conventionally structured, models may converge. When the disclosure shifts, becomes more complex, or frames strategy less straightforwardly, model outputs diverge.

That has a direct consequence for financial workflows. The model that works acceptably on mega-cap technology filings may behave differently on smaller issuers, distressed firms, foreign issuers, banks, insurers, biotech companies, or companies with messy segment reporting. Item 1 for Microsoft is not the same problem as Item 1 for a small-cap company trying to explain three pivots, two acquisitions, and a going-concern paragraph wearing a polite hat.

Behavioural diagnostics should therefore be included in deployment validation. A team should test not only average answer quality, but also:

This is where LLM governance becomes less glamorous and more useful.

The paper’s figures and tables are evidence layers, not separate theses

The study includes several result components. They should not be treated equally.

Figure 2, the overview of the explainable evaluation framework, is mainly an implementation detail. It explains the pipeline: data, prompts, model outputs, human review, automated metrics, and behavioural diagnostics. Useful, but not a result.

Tables 2 and 3 are robustness checks on the human evaluation. They report Krippendorff’s alpha and ICC. Their purpose is to test whether human ratings are reliable enough to support strong claims. The answer is: only cautiously. They do not destroy the evaluation, but they force humility.

Table 4 and Figure 3 provide the main human-evaluation evidence. They show the average scores by model and criterion. Their purpose is to compare perceived qualitative answer quality.

Tables 5 and 6 provide the main automated-metric evidence. They show that lexical and semantic metrics produce different views of model quality. Their purpose is not to replace human review, but to reveal output properties humans may not consistently quantify.

Table 7 gives company-level win rates. This is comparative behavioural evidence. It supports the idea that model performance varies by company context, though it should not be read as a universal ranking.

Table 8 and Table 9, together with Figures 4 and 5, support the stability argument. They show that model agreement varies across model pairs, companies, and years. Their purpose is diagnostic: they help reveal when the same disclosure task induces convergence or dispersion across models.

This distinction matters because businesses often overread charts. A plot of cosine similarity is not a governance policy. A table of averages is not a deployment decision. The evidence becomes useful only when each component is assigned the right job.

What Cognaptus infers for business use

The paper directly shows that, in this controlled pilot, five LLMs behave differently when answering interpretive questions about 10-K Business sections. It also directly shows that evaluation lens changes the apparent ranking, that human reviewers disagree substantially, and that model agreement varies by company and year.

Cognaptus infers three practical lessons from that evidence.

First, LLM selection for financial disclosure analysis should be task-specific. A model that is good for concise executive summaries may not be best for source-faithful regulatory phrasing. A model that performs well on semantic similarity may not be the one business users prefer. The right evaluation begins with the workflow: analyst memo, diligence triage, covenant review, investor-relations briefing, audit support, regulatory scanning, or board-pack preparation.

Second, evaluation should be multidimensional by design. A serious test suite should include human review, lexical checks, semantic similarity, factuality checks, abstention behaviour, source-grounding tests, and stability diagnostics. If that sounds like work, good. The alternative is letting a vendor demo decide your control environment.

Third, model monitoring should continue after deployment. The paper’s one-time snapshot is a limitation, but it also points to a real production issue: proprietary LLMs change. Interfaces change. Hidden routing changes. Default behaviours change. A financial workflow that depends on stable interpretation should include periodic regression tests on a fixed benchmark set of internal documents.

A practical governance pattern would look like this:

This is not about making LLMs sound more financial. They already sound financial. That was never the hard part.

The hard part is making them boringly dependable.

Where the result should not be overextended

The study is useful precisely because it is controlled. It is also limited for the same reason.

The sample is small: 21 documents from seven unusually large technology companies. These firms produce detailed, high-quality disclosures. Their Business sections are not representative of smaller companies, non-tech industries, heavily regulated financial institutions, distressed issuers, or firms with sparse disclosure practices.

The annotators were trained and calibrated, but they had engineering and NLP backgrounds rather than specialist financial-analyst backgrounds. That matters. Professional analysts may judge omission, emphasis, and inferential discipline differently from technically trained reviewers.

The prompt space is narrow. The study uses one main prompt template, a uniform truncation strategy of approximately 3,000–3,500 tokens, and deterministic or default web-interface settings depending on model access. Different prompts, retrieval methods, chunking strategies, or document-segmentation approaches could change the results.

The model snapshot is time-bound. Web-interface models are moving targets. A result from one access window is not a warranty.

Finally, the study evaluates Item 1 Business sections, not full 10-K analysis. It does not deeply test financial statements, footnotes, MD&A, risk factors, XBRL data, or cross-document consistency. Those are separate operational problems. Some of them are nastier. Naturally.

The better question is not “Which LLM reads filings best?”

The paper’s strongest contribution is not a leaderboard. It is the demonstration that financial-report interpretation is multi-dimensional, and LLM evaluation must be multi-dimensional too.

A model can be preferred by humans, close to reference wording, semantically aligned, and behaviourally stable — but not necessarily all at once. That is the point. Financial language is not just text. It is evidence, positioning, omission, strategy, risk, and liability compressed into prose that has survived lawyers, auditors, executives, and investor-relations teams. Expecting one generic benchmark score to capture that is optimistic. Charming, but optimistic.

For operators, the takeaway is practical. Build model evaluation around the actual decision environment. Decide what failure looks like. Test for it. Re-test after model updates. Keep humans in the loop where judgement, liability, or materiality is involved.

Numbers do not speak for themselves. Neither do models. Both need interpretation, and in finance, interpretation needs controls.

Cognaptus: Automate the Present, Incubate the Future.