When More Context Hurts: RareDx Treats Knowledge Use as a Governed System Component

TL;DR for operators

Giving a decision-support model access to more knowledge is not the same as making it use knowledge well. In the controlled comparisons reported by RareDx, static retrieval can perform worse than direct inference, phenotype-aware structured retrieval can materially improve diagnostic coverage, and a validation-selected router can outperform using one inference strategy everywhere.

RareDx¹ is therefore more useful as a systems-design study than as another argument for larger retrieval pipelines. Its main contribution is a controlled harness that separates the model from evidence acquisition, structured reasoning, aggregation, normalization, and routing. Its post-training contribution follows the same logic: reward partial diagnostic progress, but only after mapping outputs to recognized medical entities and constraining obvious reward shortcuts.

For clinical-AI operators, the decision changes from “How much context can we add?” to “Which evidence path should be available, under what conditions, with which validation and output controls?” The paper supports that shift across a retrospective eight-source benchmark. It does not establish clinical safety, calibration, or prospective diagnostic effectiveness.

More evidence can make the ranking worse

The familiar product instinct is straightforward: if a diagnosis is knowledge-intensive, give the model more external knowledge. That intuition becomes risky when retrieval introduces plausible but distracting candidates.

RareDx makes this visible by holding important parts of the evaluation contract constant. Heterogeneous case records are normalized into the same Top-10 disease-ranking task, predictions pass through the same disease canonicalization process, and alternative inference strategies operate over a shared medical knowledge layer. The point of this setup is attribution: performance differences can be associated more cleanly with how knowledge is accessed rather than with a different parser, output format, or scoring rule.

The resulting pattern is not monotonic. The package reports that static RAG underperforms Direct inference. In a fixed-backbone component ablation, Adaptive ReAct raises macro Hit@10 from 31.27% to 33.53% while slightly lowering Hit@1 from 17.03% to 16.49%. On Phenopackets, generic dense gene retrieval produces only a small Hit@10 increase over Direct, from 27.5% to 28.4%. Replacing it with phenotype-structured HPO-Resnik retrieval raises Hit@10 to 36.3% and gene recall@8 from 15.3% to 69.4%.

Those are ablation results rather than the paper’s main headline benchmark, but they isolate an operationally consequential mechanism. Retrieval quality depends on representation and selection policy, not merely on adding documents to the prompt.

RareDx-Harness turns inference strategy into a controllable variable

The first architectural contribution is RareDx-Harness: Direct inference, static RAG, adaptive ReAct retrieval, structured phenotype-to-gene-to-disease reasoning, reciprocal-rank aggregation, and routing are evaluated within one diagnostic contract.

That separation matters because otherwise a stronger “agent” can hide several simultaneous changes. A system may use a different backbone, retrieve more evidence, sample more candidates, normalize outputs differently, and call additional tools. A higher final score then says little about which intervention paid for the improvement.

RareDx instead shows that the inference path itself can be governed. Its stricter routing audit selects between Direct and a Diagnostic Audit strategy using only an approximately 20% development partition for each dataset, then evaluates the frozen choice on the remaining held-out cases. Under that protocol, the 9B system reaches 37.34 macro Hit@10 versus 30.54 for its matched Direct predictions, a 6.80-point gain. The 27B system reaches 38.13 versus 29.83 for Direct.

This controlled routing result deserves more weight than the archived exploratory headline route. The latter reports 38.34 Hit@10 for RareDx-9B and 40.76 for RareDx-27B, but its route configuration was selected during exploratory benchmark-level development. It is also important that the routed 9B pipeline can invoke a 35B diagnostic audit model. The result therefore measures a routed system, not pure 9B inference.

The business inference is broader than rare-disease diagnosis: if different evidence procedures fail on different data regimes, routing policy becomes part of model governance. It should be developed and validated like a decision rule, not treated as invisible orchestration.

Dense rewards need hard boundaries

The training problem has a similar structure. Exact-match diagnostic rewards are sparse: a clinically related disease may receive no useful learning signal. But unconstrained semantic similarity creates another failure mode—a model can receive credit for fabricated names, loose approximations, or an excessively long differential.

RareDx-KGPO addresses that by first projecting generated disease strings into a canonical disease space. It then supplies bounded partial credit from four channels: curated graded relevance, ontology proximity, biomedical name similarity, and phenotype-profile consistency. For each candidate, the system takes the strongest supported channel rather than simply adding correlated signals. Rank-sensitive weighting favors supported diagnoses placed nearer the top.

The crucial contribution is not a new policy-gradient estimator. The underlying optimization remains group-relative. The novelty claimed by the paper is the medical supervision and its safeguards.

The reward audit shows why those safeguards are not cosmetic. Removing the phenotype-graph channel reduces the mean reward for nearest phenotype neighbors from 0.316 to 0.008. More strikingly, removing the canonical vocabulary gate causes 82.5% of constructed pseudo-diseases to receive positive reward; with the gate, none do.

So the design pattern is not merely “use richer rewards.” It is “provide richer partial credit only inside a constrained output space.” For regulated generation tasks, that distinction can determine whether dense supervision teaches useful structure or merely creates more exploitable scoring surface.

The experiments answer different questions

One supplementary result also makes the compute trade-off explicit. On 128 development cases, eight extra sampled lists raise 27B Hit@10 from 34.38% to 46.09%, but mean generated output tokens rise from 173 to 1,556. That is a coverage-versus-generation-budget ablation, not a held-out test result.

Similarly, the reported direct-generation throughput advantage of the 9B model—45.5 cases per second versus 18.4 for 27B—excludes retrieval, tool latency, merging, normalization, and scoring. It is useful for model-serving comparison, not an end-to-end latency estimate for the full Harness.

What clinical-AI teams can govern separately

Cognaptus’s inference from the paper is that clinical-AI architecture benefits from separating at least four decisions that are often bundled together: what the backbone already knows, what external evidence it may access, how candidate outputs are normalized and verified, and when a more expensive reasoning or audit path is invoked.

That decomposition changes procurement and engineering choices. A team considering a larger backbone should test whether the observed error actually comes from insufficient parametric knowledge. A team expanding RAG should verify that retrieved evidence improves ranking under the target case distribution rather than assuming context volume is beneficial. A team using reinforcement learning should audit what receives partial credit before optimizing against that reward. A team adding routers should freeze selection rules on development data and evaluate them independently on held-out cases.

These are governable system components with different failure modes and cost profiles. RareDx’s 50-case RAMEDIS audit illustrates the point sharply: the 9B full Harness reaches 14/38/46 Hit@1/5/10 versus 2/12/28 for direct Qwen3.8-27B under matched prompts, token budget, parser, and normalizer. The comparison does not establish that 9B models are generally preferable. It shows that system design can dominate backbone size in a controlled diagnostic setting.

The boundary is clinical deployment

The evidence base is substantial for a retrospective systems paper: 4,249 benchmark cases across eight sources, shared normalization, component ablations, reward audits, overlap checks, held-out routing, and a paired case audit.

Several limits still change how the result should be used. Benchmark sources are heterogeneous, one split is small, and evaluation relies on automated matching rather than prospective clinician adjudication. Temporal disjointness is not established for the retrieval corpus. The RL validation material also shares 290 disease-plus-phenotype profiles with the 500-case Phenopackets benchmark, which can make Phenopackets-based checkpoint selection optimistic even though identifiers and case text do not match.

Most importantly, Hit@10 is a ranking metric. Better differential coverage does not establish calibrated probabilities, evidence faithfulness, subgroup robustness, acceptable harmful-omission rates, or safe autonomous diagnosis.

The paper’s more durable result is architectural: knowledge-intensive AI should not treat retrieval as an undifferentiated performance lever. Once evidence access, structured reasoning, reward design, and routing are separated, each can be measured, constrained, and governed independently. In RareDx, that control—not retrieval volume alone—is where the reported gains come from.

Cognaptus: Automate the Present, Incubate the Future.