CLOZE Encounters: When LLMs Start Editing Medical Ontologies

Hospitals already have the raw material for better medical knowledge systems. It is sitting inside discharge summaries, nursing notes, radiology reports, ECG interpretations, and all the other clinical prose that makes electronic health records look deceptively “digital” while still behaving like a very expensive filing cabinet.

The awkward part is that clinical notes are both valuable and dangerous. Valuable, because they contain granular observations that structured fields often miss. Dangerous, because they contain protected health information, idiosyncratic phrasing, and enough local context to make naïve automation look clever right up to the moment it quietly corrupts a downstream system.

The paper behind CLOZE — short for Clinical Notes Ontology Zero-shot Extension — takes that awkwardness seriously.¹ It asks whether large language models can help extract disease concepts from real clinical notes and place them into an existing medical ontology, specifically the Disease Ontology, without task-specific training data. That sounds like the kind of sentence that usually precedes an AI demo with suspiciously clean examples. This one is more interesting because the useful lesson is not “LLMs can edit ontologies.” The useful lesson is that LLMs only become useful here when their freedom is constrained by de-identification, disease-specific extraction, biomedical embedding anchors, and recursive hierarchy checks.

In other words: the model is not the system. The pipeline is the system. A small mercy, but an important one.

The real problem is not finding terms; it is placing them correctly

Medical ontologies are not simple dictionaries. A dictionary tells you that a term exists. An ontology tells you how that term relates to other terms. In healthcare, that difference matters because downstream systems often use ontological structure for retrieval, coding support, cohort selection, analytics, interoperability, and sometimes clinical decision support.

If a system merely extracts “triple-negative breast cancer” from a note, it has found a phrase. If it places that phrase under the wrong disease parent, it has created a semantic landmine. The phrase looks medical. The system looks structured. The error becomes harder to notice because it is no longer floating in free text; it has been promoted into the knowledge layer.

CLOZE therefore separates the job into two different problems:

This separation is the paper’s first serious design decision. It does not ask one model to read a note, understand all medicine, respect privacy, inspect an ontology, and update the hierarchy in one theatrical prompt. It divides the task into roles: de-identification agent, entity-extraction agent, SapBERT semantic matcher, and relation-determination agent.

That division matters because clinical ontology extension is not one cognitive act. It is a sequence of different failure modes.

A privacy failure is not the same as an extraction failure. An extraction failure is not the same as a hierarchy failure. A hierarchy failure is not the same as a synonym failure. Treating them as one magical “LLM reasoning” task is how organizations turn a research prototype into a compliance incident with nice screenshots.

CLOZE works by narrowing the LLM’s job before asking it to reason

The mechanism is easy to miss if one reads the paper as another “LLMs for healthcare” entry. CLOZE is not simply an LLM pointed at clinical text. It is a pipeline that progressively narrows the model’s degrees of freedom.

First, raw clinical notes are passed to a de-identification agent. The model is prompted to identify protected health information categories and output a structured JSON dictionary. The detected identifiers are then masked before downstream processing. The point is not that de-identification becomes solved forever. The point is that the ontology extension stage is not supposed to see raw patient identifiers.

Second, a separate entity-extraction agent reads the de-identified notes and extracts disease-only entities. This is narrower than general medical named entity recognition. It is not trying to capture every medication, symptom, procedure, lab value, and body part. The target ontology is disease-specific, so the extraction task is disease-specific.

Third, CLOZE uses SapBERT to compute biomedical embeddings for the extracted entity and candidate ontology nodes. This is the anchoring step. Instead of asking an LLM to scan an entire ontology as text, SapBERT proposes semantically plausible candidates using biomedical representation learning.

Fourth, an LLM relation-determination agent classifies the relationship between the new entity and the candidate ontology node. The relation choices are constrained: equivalence, subsetting, or neither. If the new entity is equivalent to an existing node, no new node is added. If it is a subset of the candidate, the process recurses into the candidate’s children. If no deeper child fits, the new entity is inserted as a child of the most recent suitable parent.

That recursive step is the quiet centerpiece of the paper. It converts ontology placement from a one-shot search problem into a layer-by-layer decision process.

A simplified version looks like this:

Clinical note
   ↓
PHI masking
   ↓
Disease-only entity extraction
   ↓
SapBERT finds the closest candidate node at the current ontology layer
   ↓
LLM classifies relation: equivalent / subset / neither
   ↓
If subset, move down the hierarchy and repeat
   ↓
Insert only when the search reaches the most specific defensible parent

The important word is “defensible.” CLOZE does not prove that every insertion is clinically correct. It tries to make each insertion structurally traceable: here is the extracted entity, here is the candidate parent, here is the relation classification, and here is the path through the ontology.

For enterprise use, that traceability is not a decorative feature. It is the difference between “the model said so” and “the system produced a reviewable candidate update.” Healthcare buyers tend to prefer the second one, usually because they enjoy remaining employed.

The component tests show a pipeline that is useful but not magically clean

The paper evaluates CLOZE on 100 fully annotated clinical notes from a major U.S. hospital, with each note averaging around 1,000 tokens. The seed ontology is the Disease Ontology, and the authors focus on disease-related entities.

The first set of experiments evaluates the two medical entity extraction components: de-identification and disease entity extraction. These are best read as implementation validity checks, not as the main proof of ontology quality. If these stages fail badly, the ontology extension stage is built on sand. If they perform reasonably, the more important question becomes whether the system can place entities correctly.

For de-identification, the LLaMA-3-70B-Instruct agent achieves precision of 0.60, recall of 0.68, and F1 of 0.62. It beats the tested baselines: PhysioNet, GPT-3.5, GPT-4, and LLaMA-3-8B-Instruct. The rule-based PhysioNet baseline is especially weak in this setup, with F1 of 0.28.

The win is real, but the absolute values should keep everyone sober. An F1 of 0.62 is not a license to throw raw clinical text into an automated ontology factory. It says the selected local LLM is better than the tested zero-shot alternatives on this dataset. It does not say privacy risk has disappeared. Compliance departments may now exhale gently, not go home.

For disease entity extraction, the LLM-based extractor achieves the best F1 across fuzzy matching thresholds of 60, 70, and 80. At threshold 70, for example, it reaches precision of 0.1932, recall of 0.3215, and F1 of 0.2414. Stanza has higher recall but lower precision; BioEN has much lower recall. This pattern is consistent with the task definition: the LLM is being asked to extract disease-only entities, while general biomedical NER tools are not necessarily optimized for this exact ontology-aligned target.

Again, the interpretation needs discipline. The entity-extraction scores are not enormous. They are measured against an approximate ground truth built by fuzzy matching Disease Ontology terms in the clinical notes. That makes the experiment useful for comparing methods under the paper’s setup, but it is not the same as a complete expert-labeled benchmark of all disease mentions in all clinical prose.

The component tests therefore support a modest but important claim: CLOZE has a plausible front end for privacy-aware disease concept extraction. The decisive evidence comes later, when the paper asks whether those concepts are inserted into the ontology better than baseline strategies.

The main evidence is the ablation-like ladder: LLM alone is not enough

The strongest part of the paper is the comparison among four ontology extension strategies. This comparison functions like an ablation ladder, even if it is not presented only as a formal ablation study. Each method removes or changes one key design choice:

This is where the paper becomes more than another “LLM agent” story. The results do not show that LLMs are naturally good ontology editors. They show almost the opposite: LLMs need biomedical grounding and structural procedure.

In GPT-4-based automatic evaluation, the full SapBERT + LLM-Hierarchical method produces 617 correct relationships, 158 incorrect ones, 106 “Not Sure” labels, and precision of 79.613%. The closest baseline, SapBERT + LLM-Onetime, reaches 43.113% precision. The two LLM-only methods perform worse, at 38.859% and 36.662%.

The magnitude is the story. Adding hierarchy to an LLM without biomedical embedding support does not help; in this evaluation it is slightly worse than LLM-Onetime. Adding SapBERT without recursive hierarchy helps somewhat, but not enough. The large jump comes when both pieces are combined.

That tells us something practical: the biomedical embedding model and the LLM are not interchangeable “AI components.” SapBERT is doing the semantic anchoring. The LLM is doing the relation judgment. The recursion is doing the structural search. Remove one, and the system becomes less like ontology extension and more like confident filing.

There is also a useful warning hidden in the LLM-Hierarchical result. A hierarchical prompt format alone does not guarantee hierarchical understanding. Asking the model to move layer by layer may look methodologically responsible, but if the candidate selection is poorly grounded, the process can still walk down the wrong branch with admirable procedural discipline. Bureaucracy, but make it neural.

Human evaluation supports the same mechanism, with a sharper business lesson

The paper also includes human evaluation by two biomedical terminology annotators from a nursing school affiliated with a major research university. They review sampled triplets of the form {new node, original ontology node, relationship} using four criteria: relevance, accuracy, importance, and overall satisfaction. Scores are on a 0–2 scale.

The full CLOZE design again performs best. It receives relevance of 1.05, accuracy of 2.00, importance of 1.87, and overall satisfaction of 1.87. SapBERT + LLM-Onetime is the second-best method, with overall satisfaction of 1.00. The two LLM-only methods are very weak on relevance and overall satisfaction.

This result is especially useful because it separates a system that sounds plausible from a system that produces reviewable ontology additions. The LLM-only methods can still make relationship choices, but the human scores suggest that without biomedical semantic grounding, the proposed additions are often not relevant enough to be satisfying.

For business readers, the lesson is not “use SapBERT.” SapBERT is just the concrete model used here. The broader lesson is that domain-specific retrieval or embedding should anchor model reasoning before the model is asked to make structural decisions. In healthcare, legal, finance, engineering, and other knowledge-heavy domains, the highest-value AI systems will often look less like one giant model and more like a controlled chain of specialized components.

The boring architecture is the competitive advantage. Yes, tragic. Also true.

What the paper directly shows, and what business teams may infer

The paper directly shows that, on a small but real clinical-note dataset, a zero-shot pipeline combining local LLaMA-3-70B agents, SapBERT matching, and recursive hierarchy traversal outperforms tested baseline approaches for disease ontology extension. It also shows that LLM-based de-identification and disease-only extraction can outperform selected zero-shot and biomedical NLP baselines in this setup.

That is the direct claim.

The business inference is narrower but valuable: clinical organizations and healthcare AI vendors may be able to use similar pipelines to generate candidate ontology updates from unstructured clinical notes. These candidates could improve internal knowledge graphs, clinical search, coding support, cohort discovery, and interoperability mapping. The emphasis should be on candidate updates. A candidate can be reviewed, audited, accepted, rejected, and monitored. An automated ontology mutation that quietly enters production is a different animal, and it should not be fed after midnight.

A practical enterprise workflow might look like this:

This is where the ROI conversation becomes more serious. The value is not that the system replaces ontology teams. The value is that it can reduce the cost of finding, proposing, and triaging candidate updates from messy clinical text. Human experts then spend more time reviewing structured candidates and less time hunting through narrative records like archaeologists with medical degrees.

That distinction matters because ontology work is not just a technical bottleneck. It is a governance bottleneck. If an AI system accelerates candidate generation but makes validation harder, it has not improved the operation. It has merely moved the mess from unstructured notes into the knowledge base.

The paper’s limitations are not footnotes; they define the deployment boundary

The paper is careful enough to provide useful boundaries, and business readers should not round them away.

First, the dataset contains 100 annotated clinical notes. That is understandable given the privacy and annotation burden, but it limits generalization. A hospital’s note style, specialty mix, documentation templates, abbreviations, and local naming habits can differ sharply from another’s. CLOZE may travel well as an architecture, but its performance should not be assumed portable without local validation.

Second, the extraction target is disease-related entities. The paper does not show equivalent performance for drugs, procedures, lab abnormalities, social determinants, symptoms, adverse events, or multi-relation clinical concepts. Disease Ontology extension is an important task, but it is not the entire hospital knowledge graph.

Third, the entity-extraction evaluation uses approximate ground truth based on fuzzy matching against Disease Ontology terms. This is reasonable for a difficult setting, but it means the numbers should be treated as comparative evidence under a constructed benchmark, not as a final measure of clinical entity recognition quality.

Fourth, the ontology extension evaluation relies partly on GPT-4 automatic assessment because ground truth placements are unavailable. The paper supplements this with human evaluation, which is important, but the automatic precision table still reflects model-judged correctness rather than a gold-standard ontology curation process.

Fifth, the de-identification result is better than baselines but not perfect. This matters because the paper’s privacy story depends heavily on de-identification and local deployment. In production, a healthcare organization would still need independent privacy testing, secure inference infrastructure, access control, logging, and clear handling of residual PHI risk.

These limitations do not invalidate the work. They make the correct use case clearer: CLOZE is best understood as a prototype architecture for privacy-aware, human-reviewable ontology extension from clinical notes. It is not a turnkey machine for autonomous clinical knowledge governance.

The deeper lesson is controlled agency, not medical magic

The fashionable reading of this paper would be: “LLMs can update medical ontologies from clinical notes.” That is the least useful reading.

The better reading is: LLMs become useful in medical ontology work when they are assigned narrow jobs inside a structure that compensates for their weaknesses. They de-identify under a schema. They extract only disease concepts. They classify constrained relationships. They are anchored by a biomedical embedding model. They move through the ontology recursively instead of hallucinating a global placement from flattened text.

That is why the mechanism-first view matters. CLOZE’s contribution is not just its result table. It is the architectural argument behind the table: ontology extension is a structured decision process, and model intelligence needs to be wrapped in domain grounding, procedural search, and validation points.

For healthcare AI buyers, this suggests a more mature procurement question. Do not ask only, “Which model are you using?” Ask:

This is also where CLOZE connects to a broader enterprise AI pattern. In serious domains, the useful systems are not the ones that let LLMs “think freely.” They are the ones that force LLMs to think inside operational scaffolding. The scaffolding is not a limitation. It is the product.

CLOZE shows a credible version of that pattern for medical ontologies. It starts from messy clinical notes, protects them imperfectly but explicitly, extracts disease concepts, anchors them in biomedical semantics, and uses recursive relation checks to propose ontology insertions. The result is not an autonomous curator. It is a more scalable assistant for a task that has been too manual, too slow, and too dependent on scarce expert labor.

That may sound less glamorous than “AI edits medicine.” Good. Glamour is cheap. Ontology corruption is expensive.

The future of medical knowledge systems will not be built by asking a general model to rewrite the hospital’s conceptual map in one prompt. It will be built by pipelines that know when to extract, when to match, when to reason, when to recurse, and when to stop for human review. CLOZE is one such pipeline. Not the final answer, but a useful signpost.

Cognaptus: Automate the Present, Incubate the Future.