What Happens in Backtests… Misleads in Live Trades

TL;DR for operators

A beautiful backtest can still be a lie. Not because the model is malicious, obviously; spreadsheets have not yet formed a union. The problem is simpler and more expensive: a model can fit past data while misrepresenting the thing you actually care about.

Charles Rathkopf’s paper on hallucination and reliability in scientific generative AI gives operators a useful way to think about this problem.¹ It argues that hallucination should not be defined mainly as deviation from training data. In science, and in business domains that behave like science, the real question is whether an output misrepresents the target phenomenon: a protein, a weather system, a molecule, a patient, a market, a factory, a supply chain.

That shift matters because the obvious fix—filter out anything too far from the training distribution—can destroy the whole point of using generative AI. Discovery lives near the edge of what the data already says. If you filter every unfamiliar output, you do not get reliability. You get a very expensive photocopier with a better user interface.

The paper’s mechanism-first lesson is this: hallucination is hard to eliminate, but it can be converted into bounded risk when generative models are embedded in mature workflows. AlphaFold does this through theory-guided training and confidence-based error screening. GenCast does it through physically disciplined data, probabilistic forecasting, and ensemble uncertainty. Neither becomes reliable because its inner parameters are magically transparent. Reliability comes from the workflow around the model.

For business use, the lesson is not “trust AI in science.” It is narrower and more useful: trust is operationally constructed. The model needs domain constraints, calibration, confidence thresholds, validation loops, expert interpretation, and consequences for low-confidence output. Where those are absent, “AI-driven discovery” becomes a polite phrase for “we asked a stochastic system to improvise and then nodded gravely.”

The dangerous mistake is treating the dataset as the world

Most AI reliability conversations begin with the dataset. Was the model trained on enough examples? Are the examples representative? Does the output fall inside the learned distribution? These are reasonable questions. They are also insufficient.

Rathkopf’s central correction is that scientific inference is not about fidelity to data as such. Data is constructed through measurement, instrumentation, selection, modelling, and experimental convention. The training set is not the phenomenon. It is a trace of the phenomenon, processed through human and technical machinery.

That distinction sounds philosophical because it is. It is also operationally brutal. A trading backtest is not the market. A claims dataset is not patient health. A historical demand curve is not future customer behaviour. A satellite-derived weather reanalysis is not the atmosphere itself. A protein database is not the full biochemical search space.

The paper’s phenomenon-centric view says that hallucination should be assessed by the output’s relationship to the target system. A generated output is dangerous when it misrepresents the target under the interpretive standards of the domain, is not a deliberate idealisation, and arises from the model’s generative activity rather than merely inheriting an error from the data.

That definition is more demanding than the usual “the model made something up” complaint. It also avoids the opposite mistake: calling every novel output a hallucination. Novelty is not the enemy. Unsupported confidence is.

This is where the article’s title earns its keep. Backtests mislead live trades when the evaluation target quietly changes. The model looks good against historical traces but fails against the live phenomenon. In Rathkopf’s terms, the model has achieved statistical resemblance without reliable representation. A small distinction, right up until it becomes a large invoice.

Why hallucination is not a bug that scaling politely removes

The paper treats hallucination as structurally likely in generative AI, especially in scientific settings where models produce complex, high-dimensional outputs.

A classifier chooses among labels. A generative model produces structured artefacts: molecular conformations, weather fields, medical images, protein structures, text, designs, trajectories. The output space is enormous. A 100-amino-acid protein already implies a vast combinatorial search space before conformational variants enter the party. Meteorological systems involve many variables across space and time. Even large datasets cover only a small fraction of what the model may be asked to generate.

The mechanism is not mysterious. Generative models compress sparse observations into internal representations, then sample from those representations. In high-dimensional spaces, real data often lies on lower-dimensional manifolds. When a model interpolates between such regions, it may generate outputs that are locally plausible but globally wrong.

That last phrase is the killer: locally plausible, globally wrong.

A generated protein can contain chemically plausible fragments while forming an unstable structure. A weather forecast can capture regional patterns while violating broader physical coherence. A language model can write graceful paragraphs while quietly losing the plot. Elegant local texture is cheap. Global fidelity is where the bill arrives.

The paper also discusses diffusion-model work by Aithal and colleagues as a useful contrast. Their experiments use synthetic settings where the data-generating structure is known, such as mixtures of separated Gaussians and constrained binary grids. The likely purpose of those experiments is not to prove a second thesis about scientific AI; it is to isolate a mechanism of hallucination in diffusion models. In those settings, filtering low-density samples can remove many hallucinations. Rathkopf reports that the proposed method removes 95% of hallucinatory samples while retaining most in-distribution outputs.

That is good engineering for the experimental setup. It is not a general philosophy of scientific discovery.

In real scientific and business environments, an output that falls between known modes can be either a mistake or the first hint that the known modes were incomplete. If the governance rule is “discard anything unfamiliar,” then the model becomes safe in the way a locked laboratory is safe: nothing explodes because nothing happens.

The first mechanism: theory-guided training narrows the space of nonsense

The first reliability mechanism in the paper is theory-guided training. This is the boring phrase for a powerful idea: do not ask a neural network to rediscover every constraint of the world from examples alone.

AlphaFold is the central case. AlphaFold 3 uses a generative diffusion component to predict biomolecular structures and interactions. That generative flexibility increases hallucination risk, because the model may produce plausible-looking structures where no stable structure exists. The mitigation is not blind faith in model scale. It is discipline imposed by structural biology.

AlphaFold is trained on the Protein Data Bank, a curated repository of experimentally validated molecular structures. It also incorporates violation losses that penalise physically unrealistic outputs: steric clashes, implausible bond lengths, invalid torsional angles, and other violations of molecular geometry. The model is being told, in effect, that some attractive mathematical possibilities are chemically stupid. A useful distinction, and one that more AI products might consider.

These constraints matter because they come from knowledge partly independent of the training distribution. The Protein Data Bank reflects accumulated experimental work. Physical constraints reflect established chemistry and molecular physics. When those sources converge, the model’s output is no longer just a statistical echo of stored examples. It becomes a theoretically disciplined inference about a target phenomenon.

This is where many business AI deployments are weaker than their pitch decks suggest. They have data. They have models. They may even have dashboards with gradients and tasteful rounded corners. What they lack is explicit domain theory embedded into training, scoring, filtering, and use.

For a credit model, what are the economic constraints? For a supply-chain model, what physical and contractual constraints are impossible to violate? For a trading model, what market microstructure assumptions must hold before a backtest is even interpretable? For a drug-discovery model, what biochemical constraints make an output synthesis-worthy rather than merely molecule-shaped?

Without these constraints, the model is not “creative.” It is under-parented.

The second mechanism: confidence screening prevents weak outputs from becoming beliefs

Theory-guided training reduces nonsense. It does not abolish it. The second mechanism is confidence-based error screening.

In AlphaFold, this includes residue-level reliability scores such as pLDDT, which help users identify which regions of a predicted structure are stable enough to support inference. The key is that these scores are not simple measures of closeness to training examples. They reflect the model’s estimate of local reliability, informed by patterns of agreement and calibration against validated structures.

The operational value is obvious: the model does not merely output “a structure.” It outputs a structure with signals about where interpretation should stop.

That is the part many AI systems still refuse to learn. They produce one polished answer in the tone of a management consultant who has never been wrong, merely “misaligned with emerging stakeholder realities.” Scientific reliability requires the opposite posture: show where the model is strong, where it is unstable, and where the user should not build a decision.

The paper’s discussion of intrinsically disordered regions is especially useful. Some protein regions are environmentally contingent and cannot be represented as one stable structure. A dangerous model would hallucinate a crisp structure anyway. A better workflow returns low-confidence, flexible representations that signal: do not over-interpret this. The output still exists, but the workflow changes its epistemic status.

This is the difference between generating information and authorising belief.

For operators, the lesson is not merely to ask for confidence scores. It is to make confidence operational. Low-confidence outputs should trigger different actions: discard, escalate, validate, delay, price differently, restrict use, or route to an expert. A confidence score that sits decoratively beside an answer is not governance. It is wallpaper.

GenCast shows the same mechanism at planetary scale

AlphaFold operates in molecular biology. GenCast operates in weather forecasting. The domains are wildly different, which is why the comparison is useful.

Weather is chaotic. Small initial differences can expand rapidly. Traditional numerical weather prediction relies on solving physical equations, which is powerful but computationally expensive, especially for rare, high-impact events. GenCast offers a diffusion-based generative approach trained on ERA5, a reanalysis dataset that combines observations with physics-based modelling to reconstruct coherent atmospheric states.

Again, the model is not trained on raw chaos scraped from the sky. It is trained on a theoretically curated representation of the atmosphere. That matters.

The paper argues that GenCast’s reliability comes from the same broad pattern as AlphaFold: theory-informed data, physically meaningful constraints, and uncertainty-aware interpretation. GenCast does not produce one deterministic prophecy from Mount GPU. It generates ensembles of plausible forecasts. Disagreement across ensemble members becomes evidence about uncertainty.

Here, the mechanism is ensemble-based uncertainty estimation. If stochastic runs diverge, the forecast region is unstable. If patterns recur across runs, the model offers stronger support for action. The spread-skill ratio then helps assess whether ensemble dispersion matches actual forecast error. A ratio near 1 suggests that uncertainty is calibrated rather than theatrically humble or dangerously overconfident.

The likely purpose of this GenCast evidence in Rathkopf’s paper is main evidence by contrastive extension. AlphaFold shows the mechanism in structural biology, where the target is a molecular structure. GenCast shows that a similar reliability pattern can operate in a dynamic system, where uncertainty is not a nuisance but the object being modelled.

For business readers, this is more useful than a simple “AI beats traditional forecasts” headline. The interesting part is not that generative AI can forecast. The interesting part is that it can expose the shape of its own uncertainty in a way the surrounding workflow can use.

The evidence map: what the paper uses, and what it does not prove

Rathkopf’s paper is not a benchmark paper with a new model, ablation table, and a small forest of GPU-hours quietly sacrificed to reviewer two. Its contribution is conceptual and mechanistic. It explains how generative AI can support reliable scientific inference despite hallucination and opacity.

That means the evidence has to be read correctly.

This evidence map matters because the paper’s business value is easy to misread. It is not saying “AlphaFold and GenCast work, therefore AI is reliable.” That would be the kind of conclusion one expects from a conference booth, not an argument.

The paper is saying something more conditional: generative AI can support reliable inference when embedded in workflows that connect outputs to target phenomena through theory, calibration, screening, and validation.

The unit of reliability is not the model. It is the model-in-workflow.

The business translation: stop buying models and start designing inference processes

The direct claim of the paper is about scientific generative AI. The business inference is broader but must be handled carefully.

Cognaptus’ reading is that any high-stakes AI product should be assessed through the same four-part question:

1. What is the target phenomenon?
2. What domain theory constrains valid outputs?
3. What uncertainty signals decide whether outputs are usable?
4. What validation loop catches errors before they become decisions?

If a vendor cannot answer those questions, they are not selling reliability. They are selling fluent surface behaviour with an invoice.

This framework applies differently across domains.

The finance row is where the title bites. A backtest is a dataset-shaped argument about a market. It may encode some truth. It may also encode survivorship bias, look-ahead leakage, liquidity fantasies, cost omissions, regime dependence, and the charming assumption that live execution politely resembles historical candles.

A data-centric model governance process asks: did the strategy perform well on the historical sample?

A phenomenon-centric process asks: does this output still represent the live trading environment we are about to enter?

Those are not the same question. Many expensive mistakes begin by pretending they are.

Reliability is cheaper when uncertainty is allowed to say “no”

The most practical lesson in the paper is that reliability depends on refusal. Not theatrical refusal. Operational refusal.

The model must have mechanisms for saying: this output is not stable enough, not physically plausible enough, not sufficiently calibrated, not synthesis-worthy, not trade-worthy, not decision-grade. The organisation must then respect that signal.

This is where governance becomes design rather than documentation. A good workflow defines what happens when the model is uncertain:

This is why “human in the loop” is too vague to be useful. Which human? At what threshold? With what evidence? Able to override what? Required to document which decision? Feeding back into which system?

Without those answers, the loop is ornamental. Very popular in slide decks. Less popular in court.

Where the argument stops

The paper is appropriately sharp about its own boundary. AlphaFold and GenCast are not proof that generative AI can be made reliable everywhere. They are proof that reliability is possible under favourable epistemic conditions.

Those conditions are demanding.

First, the domain needs mature theory. AlphaFold can use molecular geometry because chemistry has done a great deal of homework. GenCast can use atmospheric physics because weather is governed by quantitative laws and observed through dense instrumentation. In weaker-theory domains, the model has less external structure to lean on.

Second, the domain needs validation data. Confidence scores are only useful if they have been calibrated against something. Ensemble uncertainty only becomes meaningful when dispersion can be compared with actual error. If validation arrives years later, or never arrives cleanly, uncertainty estimates become much harder to trust.

Third, the organisation needs operational discipline. A threshold that nobody follows is not a threshold. A validation loop that never changes the model is not a loop. A confidence score that sales teams can override because the demo must look smooth is not confidence. It is theatre with decimals.

Fourth, the target phenomenon must be stable enough for the workflow to learn from past failures. Weather is chaotic but physically structured. Markets are not only stochastic but adaptive: once a strategy is deployed, other actors may respond, liquidity may vanish, and the relationship between historical pattern and future payoff may decay. This does not make AI useless in finance. It makes data-centric confidence especially dangerous.

The right conclusion is not pessimism. It is selectivity.

Use generative AI where you can build the reliability scaffolding. Be cautious where you cannot. And be deeply suspicious of anyone who treats “we trained on a lot of data” as a substitute for understanding what the outputs are supposed to represent.

The operator’s checklist for phenomenon-centric AI reliability

A practical review of a high-stakes generative AI system should not begin with the model architecture. Begin with the inference workflow.

Ask these questions before procurement, deployment, or scale-up:

That last answer—“it generalises well”—should make any operator reach for coffee, legal counsel, or both.

The conclusion: reliability is not resemblance

Rathkopf’s paper gives a clean way to diagnose a messy problem. Generative AI hallucination is not simply a matter of straying from training data. In science, and in serious business applications, the relevant standard is the target phenomenon. Does the output support a reliable inference about the thing we care about, or does it merely resemble the traces we happened to collect?

AlphaFold and GenCast show how hallucination risk can be managed without pretending the model is transparent. Their reliability comes from theory-guided training, confidence-based screening, ensemble uncertainty, domain expertise, and validation workflows. They do not eliminate error. They make error visible enough, bounded enough, and operationally manageable enough to support action.

That is the lesson for operators. Do not ask whether a model “hallucinates” in the abstract. Ask whether your workflow can detect, interpret, filter, validate, and learn from the hallucinations that matter.

Backtests mislead live trades when statistical resemblance masquerades as representation. Scientific AI fails the same way. The cure is not less ambition. It is better epistemic plumbing.

Unromantic, yes. But plumbing is usually what separates civilisation from flooding.

Cognaptus: Automate the Present, Incubate the Future.