TL;DR for operators
An analyst must combine financial ratios, management commentary, risk reports, limited research time, and an auditable recommendation. Large language models can summarize those materials fluently, but the harder question is whether prompting alone can make them apply evidence weights and decision rules consistently.
CRISTAL separates the work. An LLM converts qualitative reports into structured indicators, while explicit probabilistic code records how each indicator changes the probability of each company category, updates those probabilities, prioritizes additional evidence, and produces the final classification.
On the paper’s synthetic asset-classification benchmark, CRISTAL achieved 0.88 accuracy and 0.80 Matthews correlation coefficient—a measure of whether the system meaningfully separates all three categories rather than favoring the largest one. A prompted Llama3:70b analyst given the same likelihood information achieved 0.35 accuracy and 0.00 MCC. The difference is that CRISTAL used the likelihoods in executable inference, while the LLM received them as prompt context.
The procurement implication is architectural, not financial. Organizations should distinguish between buying a fluent AI analyst and building a hybrid decision system in which language models interpret evidence but inspectable code controls probability updates, evidence acquisition, and recommendation rules. The benchmark does not establish profitable investment performance or real-market robustness; it shows that describing a decision rule in a prompt is not equivalent to implementing it.
A fluent analyst can still lose the decision rule
A company review rarely fails because no information exists. Ratios, cash-flow statements, management commentary, regulatory reports, and industry signals are usually available in excess. The harder problem is deciding how each observation changes the recommendation, which evidence deserves attention next, and how to reproduce that judgment later.
A language model can summarize the material and produce a coherent narrative. That does not ensure that it will apply a supplied probability structure consistently. In the paper’s main comparison, both CRISTAL and the Llama3:70b analyst received the benchmark’s feature-label likelihood information. CRISTAL used those likelihoods in explicit inference; the LLM received them through prompting. Their results diverged sharply: 0.88 versus 0.35 accuracy, and 0.80 versus 0.00 MCC.
The decisive issue is where the decision rule lives. Executable probability updates behave differently from the same rule embedded in natural-language context.
CRISTAL divides interpretation from judgment
The system works with three synthetic company categories: high growth, stable, and time-bomb. Each company has hard financial indicators, qualitative reports, acquisition costs for additional evidence, and category-conditioned likelihoods. The benchmark contains 64 high-growth, 98 stable, and 38 time-bomb companies.
CRISTAL uses LLMs where language is unavoidable. A report may imply succession risk, regulatory pressure, product moat, or mismanagement without stating a standardized label. The extraction component converts that report into a red, yellow, or green indicator estimate with explanatory text.
The final judgment does not remain in the language model. CRISTAL maintains an explicit record of how each indicator state changes the probability of each company category. This is the paper’s Bayesian world model: a structured set of priors and likelihoods that produces posterior category probabilities as evidence arrives.
The division of labor is therefore precise:
| Function | Component | Audit question |
|---|---|---|
| Interpret qualitative language | LLM extractor | Was the report converted into the correct indicator state? |
| Represent feature-label relationships | Probabilistic model | Are the priors and likelihoods justified and current? |
| Update category probabilities | Bayesian inference code | Did each observation change the posterior correctly? |
| Select additional evidence | Budget policy | Was the next report worth its cost? |
| Produce the final category | Posterior decision rule | Which probability and threshold determined the output? |
This separation exposes distinct failure surfaces: extraction error, model misspecification, skipped evidence, or an incorrect decision rule. A generated rationale may compress them into one plausible paragraph.
The few-shot test evaluates updating, not memorized answers
The known-likelihood comparison could be dismissed as an advantage from hard-coding the benchmark structure. The few-shot experiment addresses that concern by asking the system to learn feature-label likelihoods from labeled examples.
CRISTAL improved quickly. With one example, it reached 0.55 accuracy and 0.29 MCC. At five examples, it reached 0.79 and 0.67; at ten, 0.83 and 0.74; and at sixty, 0.88 and 0.81. Performance dipped slightly at eighty examples to 0.86 accuracy and 0.79 MCC, so the curve is not strictly monotonic.
Most tested LLM analysts remained close to zero MCC as more examples were added. There were exceptions: GPT-4o scored 0.66 accuracy and 0.49 MCC with one example, and DeepSeek-R1 70B improved to 0.43 and 0.25 at eighty examples. Model coverage was also incomplete across shot counts. Even with those qualifications, the general pattern supports a narrow mechanism claim: explicit likelihood estimation retained more usable feature-label information than the tested in-context analyst prompts.
Showing a model examples may influence its next answer. Updating a persistent statistical relationship changes an inspectable state that can be tested, versioned, and reused.
Reversed labels test whether semantics override observation
The label-swapping experiment is a stress test, not a second headline result. The authors reversed the meanings of “high growth” and “time-bomb,” creating a mapping that conflicts with ordinary language associations.
GPT-4o and DeepSeek-R1 70B fell toward chance-level MCC under the reversed semantics. CRISTAL continued to improve from the observed likelihood data. The likely interpretation is that pretrained semantic expectations can interfere when the operating regime contradicts familiar label meaning, while an explicit model can update the mapping directly.
Pretrained knowledge is not generally harmful; open-ended analysis may depend on it. The governance risk appears when labels, policies, or regimes change and semantic familiarity resists the new decision structure. Reversed-label tests can reveal whether a system follows current evidence or reverts to prior associations.
Evidence acquisition becomes a budgeted decision
Analyst workflows also face a sequencing problem. Quantitative indicators may be available immediately, while management reports, specialist assessments, or deeper model calls consume time and money. CRISTAL processes cost-free hard indicators first, then ranks qualitative indicators by expected Bayes-factor utility and acquires them until the budget is exhausted.
In plain language, the system estimates which unseen indicator is expected to change the odds between categories most strongly. It spends the next unit of analysis on the evidence with the highest expected informational contribution, subject to synthetic costs.
A due-diligence platform could use low-cost data to establish an initial posterior, then authorize expensive review only when expected decision improvement justifies it. The paper does not provide a clean real-world estimate of savings or decision value. Because its budget, accuracy, and Bayes-optimality descriptions are not fully aligned across tables, the acquisition policy is an architectural contribution, not a validated ROI result.
The extraction layer is strong on average and uneven where language is ambiguous
The paper separately benchmarks the report-generation and indicator-extraction channel. The selected Llama3.1:8b extractor averages about 94.6% accuracy and 0.92 MCC across generator models. Several indicators approach 0.99 accuracy.
“Mismanagement” is the clear exception: 0.67 accuracy and 0.56 MCC in the detailed table. Environmental risk, product moat, or digital transformation may be expressed through relatively recognizable textual cues. Mismanagement is broader, more contextual, and easier to imply indirectly.
A reproducible Bayesian calculation can still update on a wrongly extracted state. Hybrid systems need acceptance criteria by indicator type, document source, language, sector, and reporting style—not only one aggregate score.
What an analyst platform can adopt now
Cognaptus infers three near-term design requirements from the paper.
First, preserve the probability state outside generated prose. The affected users are risk, investment, credit, and approval teams; the decision is whether an AI-produced recommendation may influence action. The condition is a task with definable hypotheses and repeatable evidence relationships. The boundary is that a misspecified model can be consistently wrong.
Second, test adaptation with deliberately unfamiliar mappings. Procurement evaluations should include reversed labels, changed thresholds, and synthetic regimes with known ground truth. These tests are more diagnostic than asking whether a model can produce a realistic research report.
Third, log evidence acquisition as part of the recommendation. A governed system should record what it observed, what it chose not to buy or review, the expected value assigned to each candidate source, and the posterior before and after each update. This turns limited attention into an auditable policy rather than an invisible prompt-side choice.
The benchmark does not establish investment performance
The evidence is confined to synthetic companies with designed priors, categorical likelihoods, generated reports, and known labels. No market returns, portfolio construction, transaction costs, live disclosures, or investment outcomes are evaluated. The benchmark’s structure also favors a method built for direct Bayesian updating.
The language channel is artificial on both sides: LLMs generate reports and an LLM extracts the intended indicators. Real filings contain strategic ambiguity, boilerplate, legal constraints, inconsistent accounting, and adversarial presentation that may produce a different error distribution.
The main comparison reports no confidence intervals, repeated-seed distributions, or comprehensive sensitivity analysis. Some model-shot combinations are missing, and claims about 100% or Bayes-optimal performance with five examples conflict with the reported five-shot accuracy of 0.79. The asserted equivalence between an 88% informativity ratio, uncertainty resolution, and classification performance is also insufficiently derived.
These limits do not erase the architectural result. They determine its status: a controlled demonstration that explicit probabilistic decision logic can outperform prompted implicit reasoning on a task designed around recoverable likelihoods.
Keep the recommendation trace outside the narrative
CRISTAL’s strongest contribution is not a claim that Bayesian programs should replace LLM analysts. It is a clearer allocation of authority. Let the language model interpret documents and propose structured evidence. Let explicit code determine how that evidence changes probabilities, which additional information deserves budget, and what rule converts the posterior into a recommendation.
For organizations evaluating AI analysts, the next step is not to accept the benchmark score as proof of financial value. It is to test the same separation on genuine disclosures, measure extraction errors by evidence type, validate calibration under regime change, and compare final decisions with human and model baselines.
A polished explanation can remain useful. It should not be the only place where the decision logic exists.
Cognaptus: Automate the Present, Incubate the Future.