Blinded by Design: When AI Stops Thinking and Starts Remembering

A name can do a suspicious amount of work.

Give an LLM a table of colorectal cancer gene candidates and ask it to rank the best drug targets. When the gene names are visible, KRAS lands at #1. The model justifies the choice with a confident reference to “proven therapeutic tractability via covalent RAS inhibitors.” Sensible enough, if the task is to combine the supplied table with the model’s accumulated biomedical knowledge.

But that was not the task.

The task was to rank candidates based on the features in the table. When the same table is shown with anonymous labels, the candidate corresponding to KRAS—now hidden as Gene_088—falls to #5. It remains important, but it no longer receives the extra boost that comes from being KRAS. More importantly, the phrase about covalent RAS inhibitors was not in the input data. It came from the model’s memory.¹

That is the paper’s cleanest example, and also the right place to begin. The problem is not that the model knows too much. Knowledge is useful. The problem is that, in many analytical workflows, the model does not tell you when it has stopped using your evidence and started using its own prior.

This paper calls the proposed intervention epistemic blinding. The phrase sounds more elaborate than the implementation. You replace entity names with anonymous codes, run the same task twice—once blinded, once unblinded—and compare the outputs. The point is not to make the blinded result automatically “better.” The point is to measure whether the identity of the entity is reshaping the decision.

For business users, that is the uncomfortable part. Many organizations are now putting LLMs inside ranking, scoring, screening, and prioritization workflows: equities, suppliers, legal cases, research papers, job applicants, clinical targets, acquisition candidates. These are exactly the workflows where names carry reputational gravity. Apple is not just a row in a spreadsheet. KRAS is not just a gene symbol. Harvard is not just a school name. McKinsey is not just a vendor.

If an LLM sees the name, the name may quietly become a feature.

The KRAS shift is not a biomedical curiosity

The KRAS example works because it separates three things that are usually tangled together.

First, the supplied data did contain meaningful signals. KRAS/Gene_088 had high mutation frequency, strong enrichment, and plausible biological importance. The blinded model did not ignore it. It ranked it #5, which is still prominent.

Second, the unblinded model added knowledge that was true but not provided. It cited therapeutic tractability through covalent RAS inhibitors. That is not hallucination in the usual sense; the statement is medically grounded. The issue is attribution. The user asked for inference from supplied features, and the model blended those features with background knowledge without marking the boundary.

Third, the ranking changed. The visible name did not merely enrich the explanation after the fact. It moved the candidate upward.

That is the paper’s central claim: named entities activate parametric priors, and those priors can reshape analytical outputs even when the prompt asks the model to reason only from the data. The effect is not a benchmark contamination problem in the narrow sense. It is not about memorized test answers inflating an evaluation score. It is closer to reputation leakage: the model has seen some entities far more often than others, and that uneven exposure becomes a hidden input.

This distinction matters because many AI governance conversations still ask the wrong first question: “Was the answer accurate?” Epistemic blinding asks a different question: “Did the model follow the analytical process we designed?”

Those are not the same. A model may be accurate because it ignored your process and used a prior. That may be acceptable in a literature review. It is much less acceptable in a controlled screening workflow where the point is to evaluate entities from comparable evidence.

What epistemic blinding actually changes

The protocol is deliberately low-tech. That is a feature, not a weakness.

The paper describes epistemic blinding as an inference-time intervention. There is no retraining, no architectural modification, and no special model access. The analyst identifies the columns that contain named entities, replaces them with stable anonymous codes, shuffles rows, preserves the quantitative features, runs matched blinded and unblinded prompts in separate fresh sessions, and then compares the outputs after de-anonymizing the blinded result.

The important operational detail is stability. If the same gene, company, applicant, or supplier appears across multiple tables, it must receive the same anonymous code everywhere. Otherwise the model loses legitimate cross-dataset structure. Blinding is not supposed to destroy the analytical task. It is supposed to remove the unnecessary identity cue while preserving the evidence the model needs.

The paper’s protocol can be summarized like this:

The paper is careful about what should and should not be blinded. Entity identifiers and proper nouns with strong training footprints are obvious candidates. Feature names, units, numeric values, and task-relevant labels usually stay visible. Some columns sit in the middle. Disease names, for example, provide essential context, but they also activate priors. Financial sectors can be analytically useful, but in combination with size and margins they can reveal the company.

This is where the protocol becomes less trivial than “find and replace.” A $3 trillion market-cap company in smartphones is not anonymous just because the ticker is hidden. A 42% mutation frequency in colorectal cancer may already point toward KRAS for a model with enough biomedical exposure. The paper calls attention to these structural leak sources and suggests mitigation by normalization, binning, or dropping highly identifying features.

That caveat is not a small footnote. In business workflows, leakage is usually where governance fails. Teams remove the obvious identifier and leave five proxy variables that reconstruct it.

The oncology experiment tests whether names move the frontier

The paper’s biology case is not just an LLM prompt experiment. It sits inside a larger target-identification system.

The system first builds a deterministic scoring function for drug target prioritization. It uses public biological data layers: somatic mutation information, GWAS associations, protein and transcriptomic embeddings, protein-function embeddings, and genetic constraint. The paper deliberately excludes literature-mined gene-disease relationships and other heavily curated sources that would reintroduce publication bias. This is an important design choice. If the input data already encodes the fame of well-studied genes, blinding the model later cannot magically recover a clean discovery process. Garbage in still has tenure.

The scoring-function stage is also blinded by design. The LLM is used as a mutation operator in an evolutionary optimization framework, but it sees only numerical feature vectors, not gene symbols or disease names. Across 13 generations and 53 candidate scoring functions, the system evolves from a naive maximum-enrichment rule to a more structured hierarchy: direct genetic evidence receives strong priority, while pathway-neighbor evidence receives lower but still meaningful weight.

This part of the paper has a different evidentiary role from the later A/B blinding test. It is mainly an implementation demonstration: the authors show that an LLM can help evolve useful scoring logic without seeing the names of genes. It supports the broader feasibility claim, not the contamination claim by itself.

The contamination evidence appears in the second stage. For four oncology indications, the deterministic model selects the top 100 candidate genes. Then matched blinded and unblinded prompts ask Claude to rank the top 20 targets based exclusively on the provided features.

The four indications were chosen to span different signal conditions:

The aggregate result is tidy but easy to misread. Blinded and unblinded top-20 lists overlap by 84% on average. In other words, blinding changes 16% of the top-20 predictions. At the same time, recovery of validated targets is identical across blinded and unblinded conditions, averaging 2.75 approved targets per indication.

That does not mean blinding is irrelevant. It means the effect appears most clearly in the frontier of novel or less-established candidates, not in the recovery of already validated biology. The model still recognizes the obvious targets under both conditions. The difference is which additional candidates get promoted when the famous names are removed from view.

This is exactly where business interpretation should be careful. If your workflow is designed to recover known winners, prior knowledge may help. If your workflow is designed to discover overlooked candidates, prior knowledge may quietly drag you back toward the obvious.

The rank shifts show fame bias, not random noise

The paper’s rank-shift examples are the real diagnostic evidence.

When names are visible, famous genes are promoted. PTEN in glioblastoma moves from #15 when blinded to #3 when unblinded. RNF43 in pancreatic adenocarcinoma moves from #20 to #6. KRAS in colorectal cancer moves from #5 to #1. In each case, the model’s justification leans on recognized biological reputation.

The reverse also appears. DPP8 in glioblastoma has the strongest triple-enrichment signal in the dataset, with ESM2=60.9, Geneformer=30.5, and ProtT5=16.6. It ranks #3 when blinded but drops to #9 when named. SCN1A in colorectal cancer, described as having the best triple-modality convergence, drops from #2 to #11. KCNH7, with triple convergence, pLI=0.91, and 1,252 neighbors, drops from #6 to #17.

That pattern is more important than any single movement. The model is not merely unstable. It is directionally unstable. Familiar entities move up when their names are revealed; obscure entities with strong supplied features move down.

The paper also identifies a useful mechanism: contamination increases when feature ambiguity increases. AML, where the signal is strong and drivers are fewer, shows 90% overlap between blinded and unblinded lists. Glioblastoma, where signals are weaker and more candidates sit near one another, shows 75% overlap and the largest single shift.

That is how many decision systems behave. When evidence is decisive, reputation has less room to operate. When evidence is ambiguous, reputation fills the gap. The model is not being dramatic; it is doing what prediction systems do. The problem is that the user may think the model is still following the spreadsheet.

The S&P 500 test makes the paper harder to dismiss

A skeptical reader might say: fine, biomedical gene symbols are special. LLMs have absorbed mountains of gene-disease literature, and oncology is packed with canonical entities. Perhaps this is a drug-discovery problem wearing a general AI label.

The paper answers that by moving to equity screening.

The financial experiment asks an LLM to rank the 20 most attractive value investments from the S&P 500 using structured fundamentals: valuation, growth, quality, and shareholder-return features. The task is deliberately conventional. No special biotech knowledge. No disease names. Just companies, tickers, and financial ratios.

Because market capitalization and sector can identify mega-cap stocks, the paper normalizes all features by sector and blinds ticker symbols as the entity column. Five random seeds are run to estimate variation.

The result: the mean top-20 overlap between blinded and unblinded rankings is 13 out of 20. The Jaccard index averages 0.48. Mean rank delta is 3.1 positions. Put more plainly, seven of the top 20 recommendations change when tickers are revealed, a 35% reshaping of the list.

The paper reports systematic directional effects here as well. Tickers such as ELV and CI are promoted when unblinded in four of five runs, while CTRA is demoted in three of five runs. This does not prove those promotions or demotions are financially wrong. That is not the claim. It shows that the visible ticker changes the recommendation even when the prompt asks the model to rank by supplied data.

This S&P 500 section has the role of an exploratory cross-domain extension. It is not a full theory of equity analysis, and it should not be read as a trading study. Its value is narrower and sharper: the same contamination mechanism appears outside biology.

That makes the paper more relevant for business workflows. A company name, vendor name, university name, fund name, court case name, or product name can carry the same hidden prior as a gene symbol or ticker. The domain changes. The failure mode travels well. Annoyingly well, one might say.

What the evidence supports—and what it does not

The paper’s evidence is strongest when interpreted as an auditability argument.

This table matters because the obvious bad article version would say: “LLMs are biased; blinding fixes it.” That is too blunt and, worse, less useful.

The better interpretation is: LLMs blend supplied evidence with memorized priors; epistemic blinding makes that blend observable under controlled comparison. Sometimes the prior may improve practical recommendations. Sometimes it may suppress novel but data-supported candidates. The protocol does not decide which case you are in. It gives you a way to find out.

That is a governance tool, not a magic filter.

The business value is process auditability, not automatic accuracy

The paper’s most transferable business lesson is not “anonymize everything.” It is “separate data adherence from outcome quality.”

Many organizations evaluate AI workflows by checking whether the final output seems plausible. Plausibility is cheap. A model can produce a plausible ranking of companies, suppliers, job applicants, or research papers precisely because it knows the reputational landscape. That is useful when the task is synthesis. It is dangerous when the task is controlled comparison.

Epistemic blinding creates an additional evaluation axis:

For Cognaptus-style automation work, this distinction is practical. Suppose a company uses an LLM agent to shortlist vendors from a procurement dataset. If the vendor names are visible, the agent may reward familiarity, market presence, or brand reputation even when the scoring rubric is supposed to emphasize price, delivery reliability, and defect rate. That may or may not be desirable. But the team should know it is happening.

The same applies to investment research. If a model ranks value stocks and recognizable tickers move upward after being revealed, the analyst has learned something about model behavior. The right response may be to keep two tracks: a data-only blinded screen for candidate discovery, and an unblinded knowledge-rich review for contextual due diligence. Blinding does not replace judgment. It prevents the first-pass screen from becoming a popularity contest with Excel columns.

In hiring, the analogy is obvious enough to be dangerous. Blind review is already familiar in human processes, but LLM workflows often reintroduce identity cues through school names, company names, awards, and publication venues. Whether those cues should be removed depends on the decision policy. The key is not moral theater. The key is alignment between the intended rubric and the information the model can use.

In legal and research workflows, entity priors may be valuable. A famous precedent or highly cited paper is famous for reasons. But if the task is to discover neglected cases or overlooked papers with strong structural relevance, unblinded LLM screening can collapse back toward canonical references. Congratulations: the agent has automated the literature review’s worst habit.

When blinding should be used—and when it should not

The paper offers a simple heuristic: if you would blind a human analyst, you should consider blinding the LLM.

That is a good starting rule, but business teams need a slightly more operational version. Blinding is most useful when three conditions hold.

First, the supplied data contains decision-relevant signal. If the table is weak, incomplete, or mostly decorative, blinding will not rescue the workflow. It will merely produce a cleaner version of a bad screen.

Second, the entities have uneven representation in the model’s training corpus. This is almost always true for public companies, universities, researchers, drugs, celebrities, legal cases, consumer brands, and well-studied genes. The question is not whether the model has priors. The question is whether those priors are relevant to the intended task.

Third, someone will act on the output. If an LLM casually summarizes famous companies, nobody needs a blinding protocol. If an LLM prioritizes which 20 candidates receive human review, funding, outreach, due diligence, or experimental validation, the ranking has resource consequences.

There are also cases where blinding is counterproductive.

Do not blind when the task is explicitly knowledge retrieval. If the goal is to ask what is known about KRAS, Apple, or a Supreme Court case, hiding the name defeats the purpose. Do not blind when names encode legitimate functional information. Chemical names, drug suffixes, and some technical nomenclatures carry semantic content. Removing them can destroy signal rather than remove bias. Do not blind when the domain context itself is necessary and no adequate replacement can preserve it.

The right design is often staged rather than absolute: blinded first-pass screening, then unblinded contextual interpretation. This preserves discovery discipline without pretending that prior knowledge has no value.

The limitations are practical, not cosmetic

The paper is refreshingly explicit that epistemic blinding does not solve the entire bias problem.

The first limitation is data bias. If the underlying dataset already reflects historical attention, publication bias, or selection bias, blinding the LLM at inference time cannot undo that. In the oncology case, the authors mitigate this by excluding literature-mined gene-disease relationships and heavily curated sources. In many business datasets, analogous cleanup will be harder. Sales records, vendor histories, customer support logs, and CRM fields often encode organizational habits long before the model sees them.

The second limitation is structural leakage. Identity can be inferred indirectly. A company’s sector, size, margins, geography, and growth profile may reveal it even without a name. A candidate’s prior employer, school, patent list, and city may reconstruct their identity. A court case’s facts may reveal the case. Blinding reduces leakage only to the extent that the remaining features do not function as a fingerprint.

The third limitation is model specificity. The experiments use Claude. Other models may show different contamination magnitudes depending on training exposure, instruction tuning, retrieval behavior, and decoding settings. The qualitative mechanism is plausible across models, but the paper does not establish universal effect sizes.

The fourth limitation is run-to-run variance. The S&P 500 experiment uses five seeds, which helps show the effect is not just one unlucky sample. The oncology experiment uses single runs per indication, so its exact percentages should not be over-read. The directional rank-shift evidence is compelling as a diagnostic pattern, but a production governance process would need repeated runs, stable prompts, and threshold rules.

The fifth limitation is the absence of ground truth for novel candidates. If blinding promotes a less famous gene or stock, we do not automatically know that the promoted entity is better. At the discovery frontier, “better” often requires downstream validation. Epistemic blinding can show that the frontier moved. It cannot prove that the new frontier is correct.

These limitations do not weaken the protocol’s business value. They define it. The output of blinding is not truth. The output is visibility into whether identity priors are influencing a process that may have been advertised as data-driven.

A small protocol with large governance implications

The paper also releases a model-agnostic implementation and a Claude Code skill. The tooling includes scripts to create blinded and unblinded prompts, reverse mappings, and compare A/B outputs using overlap, rank metrics, and optional fame-bias statistics. The Claude Code skill is meant to reduce adoption friction by making blinding a contextual action inside an agentic workflow.

That adoption point deserves attention. Many governance controls fail because they require analysts to leave the workflow. In principle, everyone agrees that controlled comparisons are good. In practice, people are busy, prompts are messy, and the spreadsheet needs to be ranked before the meeting. A protocol that can be run inside the same workflow has a better chance of becoming routine.

The deeper implication is that LLM governance will need more of these cheap diagnostic layers. Not every risk requires a new benchmark, fine-tuned model, or enterprise platform. Sometimes the useful control is almost embarrassingly simple: hide the names, run the task again, and see what changes.

The simplicity is precisely why the result is uncomfortable. If a name replacement changes 16% of oncology target recommendations or 35% of S&P 500 value rankings, then many current LLM-assisted analysis workflows are probably less controlled than they look.

Not broken. Not useless. Just contaminated in a way that teams are not measuring.

The takeaway: memory is a feature until it becomes a hidden variable

LLMs are not clean reasoning engines. They are reasoning systems wrapped around vast memory. Most of the time, that is why they are useful. The trouble begins when a workflow pretends the memory is not there.

Epistemic blinding gives teams a way to stop pretending. It does not make the model objective. It does not guarantee better decisions. It does not erase biased data, structural leakage, or stochastic variation. What it does is restore one missing audit question: did the result come from the evidence we supplied, or from the model’s prior exposure to the entity?

For discovery workflows, that question is not academic. It determines whether AI expands the search space or simply repackages the reputation hierarchy it learned during training.

And if the model is ranking your candidates mostly because it recognizes their names, then the agent is not discovering. It is remembering with confidence.

Useful, sometimes. Dangerous, often. Auditable, now.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast