No CIG, Still Checking: When Medical Guidelines Become Executable

TL;DR for operators

Most organizations do not have a compliance problem because nobody wrote the rules down. They have a compliance problem because the rules exist in prose, the operational evidence exists in messy records, and the bridge between the two is usually a small group of overworked experts quietly aging in a meeting room.

The paper analyzed here proposes a way to build that bridge for stroke care without first creating a full Computer-Interpretable Guideline. Instead of asking one large language model to “judge” whether care was correct — a phrase that should immediately make any hospital lawyer reach for coffee — the authors build a staged pipeline. One model extracts patient traces from discharge letters. Another extracts IF-THEN rules from textual guidelines. Another filters rules that cannot be checked against the available event log. Another converts rules into Python. A stronger model refines the generated code. A conventional Python module then computes conformance.

That sequence matters. The LLMs are not the adjudicator of care quality. They are artifact factories: trace builders, rule extractors, code generators, and code cleaners. The final check is executable. This is the difference between “AI says the hospital complied” and “a generated rule was run against an extracted event log, with clinicians validating the intermediate artifacts.” One is an oracle. The other is at least the beginning of an audit trail.

In the experiment, the system was applied to 463 anonymized discharge letters from real stroke patients treated at the neurological ward of Alessandria Hospital between 2022 and 2024. The event log averaged 47 activities per patient and a mean hospitalization duration of 10 days. NotebookLM extracted 161 guideline rules across nine categories from an Italian stroke guideline; after filtering, 50 rules remained applicable to the available event log. Across those 50 rules, the reported overall conformance was 86.1%.¹

The result is useful, but not because 86.1% is a magical compliance score. It is useful because the pipeline surfaces where the organization should look next. A rule about cerebral hemorrhage treatment reached 62% conformance and pointed to possible coordination issues between emergency care and the neurological ward. Another hyperthermia rule scored only 6%, but the authors found many “violations” were clinically sensible substitutions: bacterial infections treated with antibiotics, COVID-19 cases treated with antivirals and anti-inflammatories, brief fever episodes resolving spontaneously, or cases transferred before treatment would be visible in the ward data. Compliance analytics, meet medicine. It has context.

For business use, the lesson is wider than stroke care. LLMs become more credible in regulated operations when they convert unstructured policy and records into inspectable, executable, and reviewable artifacts. The business value is cheaper process diagnosis, not autonomous judgment. The limitation is equally important: this is one ward, one domain, one hospital dataset, unavailable patient data, a 20% expert-validated trace sample, physician correction of the rule set, and a large drop from extracted rules to checkable rules because operational records did not contain everything the guideline required. In other words, promising. Not magic. Tragic, I know.

The real bottleneck is not intelligence; it is formalization

Conformance checking sounds simple until someone tries to do it in a hospital.

In process mining, the basic idea is to compare observed behavior against prescribed behavior. The observed behavior is an event log: a trace of activities, timestamps, and cases. In healthcare, a trace might represent what happened to a patient during a hospital stay. The prescribed behavior is a guideline or process model. Then the system checks whether the trace conforms to the model.

On paper, elegant. In a hospital, less adorable.

The patient record is often narrative text. The guideline is often a long natural-language document. The process model needed for classic conformance checking is usually not sitting there in a clean machine-readable format, waving politely. Computer-Interpretable Guidelines exist as a research and engineering tradition, but building them requires knowledge elicitation, modeling, and domain expert time. That is precisely the expensive part.

This paper’s contribution is to attack that bottleneck directly. It does not say: “Let us build a better stroke model.” It says: “Can we get from discharge letters and textual guidelines to executable conformance checks without first handcrafting a full formal guideline?”

That is a much more operationally interesting question. Enterprises usually do not lack policies. They lack executable policies.

The paper’s answer is a modular LLM-orchestrated pipeline. The word “orchestrated” is doing actual work here. The authors do not merely prompt one model and hope the hospital becomes legible. They divide the task into stages, each producing an artifact that the next stage can use.

The pipeline is essentially this:

This is the mechanism-first story. The interesting part is not that an LLM touched a clinical guideline. That is now barely news. The interesting part is that the authors make LLMs produce intermediate machinery rather than final authority.

The pipeline’s final output is the Trace Conformance Indicator, or TCI. In plain terms, for a given rule, TCI is the percentage of applicable traces that conform to that rule:

The exact implementation classifies a rule applied to a trace as “not applicable,” “conformant,” or “not conformant.” That distinction is important. A rule can be irrelevant to a patient because the condition in the IF clause never occurs. That is not a violation. A violation occurs when the rule applies and the required behavior is absent, delayed, or otherwise inconsistent with the guideline.

This is where the design becomes more serious than the usual “LLM for healthcare” wallpaper. The pipeline separates extraction, filtering, formalization, refinement, and execution. Each stage narrows the problem. Each stage creates something that can be reviewed.

The LLM is not the medical judge, and that is the point

The most tempting misreading of the paper is also the most dangerous one: “The LLM judges whether care followed the guideline.”

No. That is not what happens.

The LLMs extract and transform. The conformance check is performed by generated Python rules running on event-log traces. Clinicians validate the traces and the extracted rules. The LLM is not being asked to issue a medical verdict from a discharge letter like a tiny stochastic consultant in a white coat.

This distinction matters for deployment.

A direct LLM judge would be difficult to audit. It might produce plausible explanations, but plausibility is not control. In regulated workflows, the question is not only whether the answer sounds reasonable. The question is whether an operator can inspect the evidence path: which rule was used, which trace events matched, which timestamp was used, which branch of the rule fired, and why the trace was labeled conformant or not conformant.

The paper’s generated Python example shows the practical shape of that audit path. For a mechanical thrombectomy rule, the code searches for terms related to intracranial internal carotid artery occlusion, thrombectomy, and stroke onset. It parses timestamps, computes whether the procedure occurred within 360 minutes, and returns “CONFORMANT,” “NOT CONFORMANT,” or “NOT APPLICABLE.” The code shown is partial, but its purpose is clear: the model-generated output becomes executable logic rather than prose judgment.

That does not make the system automatically safe. Generated code can be wrong. Synonym lists can miss things. Timestamps can be ambiguous. The paper’s own architecture includes a refinement stage using Gemini 3 Pro-Preview specifically to fix bugs, eliminate noise, and remove artifacts in the generated Python rules. Sensible. Machine-generated code in a medical audit pipeline should not be treated as divine scripture. It should be treated as code. Exotic concept.

The deeper business principle is this:

That last row is not a footnote. It is the difference between useful analytics and administrative vandalism.

Trace extraction turns discharge letters into operational evidence

The first technical challenge is the event log. Without it, there is nothing to compare against the guideline.

The authors use Gemini 2.5 Flash to process discharge letters and generate XES traces. XES is a standard event-log format used in process mining. The prompt instructs the model to produce a valid XML file, maintain strict chronological order, use specific tags for timestamps, activity names, and notes, and standardize activity names in Italian against a reference list.

This is not a casual prompt. It includes role prompting, system constraints, contextual information, and few-shot examples. The model receives a discharge letter and a standardized activity vocabulary. It must map variants in the clinical text to standard activity names where possible, and only create a new standard activity when no mapping exists.

The activity synonym reference list was itself learned by an LLM in an earlier step using the discharge letters, then validated by medical experts. That detail is small but important. In real operations, language variation is not a cosmetic problem. The same clinical action may be described in multiple ways. If the trace extractor treats every wording variation as a new activity, conformance checking collapses into vocabulary confetti.

In the experiment, Gemini 2.5 Flash extracted 463 event-log items from 463 anonymized discharge letters. The average trace had 47 activities, and the mean hospitalization duration was 10 days. The authors randomly selected 20% of the traces for expert validation. Physicians checked whether the trace matched the discharge letter, whether activity names were correct with respect to the synonym dictionary, and whether timestamps or temporal ordering were correct. All checked traces were judged correct and usable.

This is main evidence for feasibility, not proof of full clinical reliability.

The 20% validation sample supports the claim that the trace extraction pipeline can produce usable traces in this setting. It does not prove all 463 traces were flawless. It also does not resolve whether the same prompting strategy would work in a different hospital, specialty, language setting, documentation style, or electronic health record environment.

But as a business result, it is still meaningful. One of the hardest parts of operational analytics is converting records written for humans into event logs fit for machines. The paper demonstrates that LLMs can help do that conversion in a real hospital dataset, with expert validation on a sample. That is not a finished product. It is a credible architecture pattern.

Rule extraction is where the guideline becomes checkable

The second technical challenge is the guideline.

A clinical guideline contains recommendations, background, evidence summaries, condition-specific nuances, prevention advice, acute-care instructions, rehabilitation guidance, and content not relevant to a given ward. It is not naturally organized as a neat list of executable constraints. If it were, half of medical informatics would have had a quieter few decades.

The authors use NotebookLM to extract 161 IF-THEN rules from the Italian stroke guideline. The rules are grouped into nine categories:

The authors report that medical collaborators reviewed the extracted rules, focusing especially on the first four categories because those correspond most directly to work in the neurological ward. The rules were considered semantically accurate and clinically relevant. The LLM also usefully omitted less crucial material, such as clinical trial summaries included in the guideline to support recommendations. That is one of the underrated strengths of this task: summarization is not merely compression; it is selection of what can become an operational constraint.

But the extraction was not perfect. Three rules were incomplete relative to the original guideline. Four rules addressed subarachnoid hemorrhage diagnosis, while cerebral hemorrhage diagnosis was not covered in enough depth. The authors corrected this through a human-in-the-loop prompt refinement step. One further iteration was enough in their experiment.

This is not a weakness that ruins the paper. It is the deployment model revealing itself.

For rule extraction in regulated domains, human-in-the-loop correction is not a temporary embarrassment until the model becomes magical. It is part of the control system. Expert review makes sure the extracted rules reflect the domain, not just the model’s confidence about how guidelines usually sound. The point is to reduce expert labor, not remove expert responsibility. Removing expert responsibility is cheaper only until the lawsuit arrives, which tends to be an expensive form of user feedback.

The paper also reports a practical model-selection observation. Gemini produced only 16 rules for rule extraction, while NotebookLM produced 161. In prior work with a different event log, GPT-5 Thinking produced 28 rules. The authors therefore used NotebookLM for this stage. This is best read as an implementation detail and exploratory comparison, not a benchmark result. The models were not evaluated under a fully controlled comparative protocol in this paper. Still, it tells operators something useful: different stages may require different model behaviors, and the “best” model for one transformation may be unimpressive at another.

Filtering is not cleanup; it is the boundary between policy and available evidence

After extracting 161 rules, the pipeline filters them. Only 50 survive as applicable to the available event log.

That large drop is not a side issue. It is the moment the paper becomes operationally honest.

Many guideline rules refer to processes outside the neurological ward’s hospitalization phase, such as prevention. Others require data that were not collected or not reported in the discharge letters. Those rules may be medically valid and operationally important, but this dataset cannot check them.

This is where many compliance systems quietly lie. They report on “guideline adherence” as though the data exhaust covers the whole patient journey. It usually does not. It covers what the system captured, what fields were populated, what notes were written, and what could be extracted.

The paper’s filtering stage makes this boundary explicit. It separates three things that operators often blur:

This matters because the filtered-out rules can be more valuable than they look. They reveal where the organization’s records are insufficient for audit. In this paper, the authors explicitly note that future work may integrate discharge-letter information with other hospital data sources for completeness. That is exactly the business interpretation: conformance checking does not only measure process quality; it diagnoses data readiness.

A compliance program that cannot distinguish “we complied,” “we failed,” and “we do not collect the evidence needed to know” is not a compliance program. It is an opinion with formatting.

Code generation makes rules executable, but refinement makes them less embarrassing

The rule-coding stage converts filtered IF-THEN rules into Python scripts. Each script, when applied to a trace, returns one of three labels: not applicable, conformant, or not conformant.

This is where the architecture shifts from language processing to executable control.

Generated code is not inherently trustworthy. The paper recognizes this by adding a separate rule-refinement stage. Gemini 3 Pro-Preview is used to improve the Python rules, fix bugs, remove redundancy, and eliminate artifacts. The authors selected it because they considered it more capable than the other models used in the pipeline.

The likely purpose of this stage is implementation robustness. It is not an ablation in the formal sense; the paper does not provide a before-versus-after error rate showing how much refinement improved generated code. It is a design element, motivated by the known brittleness of generated scripts. That distinction matters. We should not overclaim that the paper proves the refinement model is necessary or optimal. It shows that the authors built a refinement step into the pipeline and used it to produce executable rules for their conformance analysis.

From an enterprise perspective, the rule-coding and refinement stages are where governance should concentrate. A generated rule can fail in several ways:

The paper’s pipeline addresses some of these through human validation and model refinement. A production system would need more. In particular, organizations would want unit tests for generated rules, versioning of guideline-derived rules, traceability back to guideline paragraphs, audit logs for model outputs, and explicit treatment of missing data. Boring? Yes. Also the part that keeps the system from becoming a compliance slot machine.

The 86.1% result is a starting point, not the story

The headline experimental result is that, across the 50 applicable rules, the overall conformance distribution was 86.1% conformant and 13.9% nonconformant.

That is main evidence for the pipeline’s ability to produce actionable conformance statistics in a real setting. It also provides evidence of generally high adherence to stroke-care guidelines at Alessandria Hospital’s neurological ward, within the scope of the rules and data available.

But the score is not the main intellectual payload. The score tells the hospital where to inspect. The inspection tells the hospital what the score means.

The authors investigate two specific rules: one with medium TCI and one with very low TCI. These are exploratory diagnostic analyses, not robustness tests. They show how the pipeline can support process review after the numerical result appears.

The first investigated rule concerns cerebral hemorrhage. If a patient has cerebral hemorrhage, treatment for high blood pressure and reversal of anticoagulant therapies should begin as soon as the patient arrives at the emergency room. This rule achieved a TCI of 62%.

The authors found that, in most nonconformant cases, therapy was delayed because it was managed only in the neurological ward rather than in the emergency room. That points to a possible coordination bottleneck between wards. In one specific case, however, the situation was more complex: the patient was initially treated as a home-accident victim, and only later was it understood that an ischemic stroke had caused the accident.

That example matters because it shows two different meanings of nonconformance. One is organizational: the process handoff may be too slow. The other is diagnostic complexity: the clinical pathway was not obvious at presentation. A useful system must help distinguish them. A crude system would merely produce a red mark and proceed to congratulate itself on being data-driven.

The second investigated rule concerns ischemic stroke and hyperthermia. If a patient has ischemic stroke and hyperthermia, pharmacological correction is indicated, preferably with paracetamol, while maintaining temperature below 37°C. This rule had a TCI of only 6%.

At first glance, that looks disastrous. Then the authors reviewed the nonconformant traces. All involved patients had hyperthermia, but treatment differed from paracetamol. In many cases, the diagnostic path identified the cause of fever. Specifically, 69% of the traces related to bacterial infections treated with antibiotics, and 6% related to COVID-19 infection treated with antivirals and anti-inflammatories. Other cases involved brief self-resolving fever, or fever close to discharge or transfer.

In other words, many apparent violations were clinically appropriate adaptations. The physicians treated the cause rather than merely lowering the symptom.

This is the paper’s most important operational lesson. Conformance scores do not replace human interpretation. They focus it.

The business translation is straightforward: use conformance scoring as triage. Do not use it as sentencing.

The architecture generalizes better than the medical result

The stroke-care result is domain-specific. The architecture is broader.

Many regulated organizations have the same basic topology:

1. Operational evidence is trapped in semi-structured or unstructured records.
2. Normative requirements are trapped in policy documents, regulations, manuals, standards, or guidelines.
3. Formal process models are incomplete, outdated, or nonexistent.
4. Expert review is expensive.
5. Management still wants a compliance dashboard, because management enjoys dashboards the way toddlers enjoy buttons.

The paper suggests a practical path: use LLMs to convert both sides of the comparison into structured artifacts.

For hospitals, that means discharge letters and clinical guidelines. For financial services, it could mean case notes and internal risk policies. For insurance, claims narratives and coverage rules. For procurement, transaction histories and vendor compliance requirements. For construction, inspection reports and safety procedures. The mapping is not automatic, but the pattern is recognizable.

The return on investment is not “fewer doctors” or “no compliance team.” That interpretation belongs in the drawer labeled “things said by people who have never operated anything.” The more credible ROI pathway is:

The important phrase is “first draft.” The system accelerates the production of audit artifacts. It does not eliminate the need to validate them.

In fact, the architecture is useful precisely because it leaves artifacts behind: traces, rule lists, filtered rule sets, Python scripts, TCI outputs. Those artifacts can be inspected, versioned, challenged, and improved. A black-box answer cannot.

The uncomfortable lesson: missing data is part of the process

The filtering stage exposes a recurring enterprise problem: organizations often cannot audit what they did because they did not record the evidence in the right place.

In the paper, only 50 of 161 extracted guideline rules remained applicable. Some exclusions were expected because prevention, secondary prevention, rehabilitation, or special-population recommendations may not belong to acute neurological ward hospitalization. But other exclusions came from missing data: the needed information was not collected or not reported in the discharge letter.

That is not merely a limitation. It is an operational diagnosis.

A hospital could respond in at least three ways:

The same applies outside medicine. If a bank’s compliance notes do not capture the decision rationale required by policy, the problem is not that the LLM cannot infer it. The problem is that inference is not evidence. If a safety inspection report omits timestamps, no model should confidently verify time-bound safety procedures. If procurement records do not link approvals to policy exceptions, an audit pipeline cannot conjure governance from vibes.

The business value of systems like this may therefore be indirect. They can reveal where an organization’s records are not audit-ready. That discovery is occasionally unpopular, which is a reliable sign that it matters.

What the paper directly shows, and what Cognaptus infers

The paper directly shows a working prototype architecture applied to real stroke-care discharge letters from one hospital ward. It shows that LLMs can extract traces, extract guideline rules, filter them, generate executable checks, and compute conformance metrics. It reports expert validation of a 20% trace sample, expert checking and correction of extracted rules, and overall conformance of 86.1% across 50 applicable rules.

It also directly shows that nonconformance needs interpretation. The cerebral hemorrhage rule suggested a possible coordination bottleneck. The hyperthermia rule showed that low conformance can reflect legitimate clinical alternatives.

Cognaptus infers a broader business pattern: LLMs are most useful in regulated process monitoring when they are used to transform messy operational and policy text into auditable intermediate artifacts. The prize is not autonomous judgment. The prize is cheaper, faster, more repeatable process diagnosis.

What remains uncertain is the portability and reliability of the approach. The paper does not establish performance across hospitals, languages, specialties, guideline types, model choices, or data systems. It does not provide a full benchmark comparing alternative LLMs under controlled conditions. It does not quantify the error rate of generated Python rules before and after refinement. It does not validate every extracted trace. It does not make patient data available, for privacy reasons, which limits external replication.

That does not make the paper weak. It makes the result appropriately scoped.

The best reading is: this is a credible design pattern with encouraging evidence in one real clinical setting. It is not a finished compliance product. Operators who cannot tolerate that distinction should not be allowed near production AI systems, or probably spreadsheets.

Deployment boundaries that matter

A production version of this architecture would need stronger controls than the research prototype. The paper itself points toward some of them, including alternative healthcare-specific LLMs, partial automation of trace/rule verification, LLM-as-a-Judge methods for preliminary validation, and integration with additional hospital data sources.

For operators, the near-term checklist is more concrete.

First, the system needs artifact lineage. Each conformance result should link back to the guideline passage, extracted rule, generated code version, trace events, timestamps, and source document. Without lineage, the score is decorative.

Second, missing data must be represented explicitly. “Not applicable” is useful, but “not checkable because data are missing” deserves its own governance pathway. Otherwise, data gaps disappear inside harmless-looking denominators.

Third, generated rules need tests. A few manually constructed traces with known expected outcomes can expose code errors before the system reviews real patients. This is not glamorous. Neither is handwashing. Both exist for reasons.

Fourth, clinical exceptions need a review taxonomy. Some nonconformance is a bottleneck. Some is diagnostic complexity. Some is justified adaptation. Some is documentation failure. Some is genuine poor practice. A conformance system that cannot distinguish these categories will teach staff to distrust it, and they will be right.

Fifth, the system should be deployed as decision support for quality assessment, not as automated punishment. The paper’s own examples show why. A 6% TCI rule looked bad until the authors examined the clinical context. Any management process that turns that number into a disciplinary dashboard before review deserves the chaos it creates.

The business value is process diagnosis, not automated virtue

The most useful thing about this paper is that it refuses, perhaps unintentionally, the laziest version of healthcare AI.

It does not claim that the model understands medicine better than physicians. It does not ask the LLM to be a judge. It does not pretend that a guideline score is the same thing as care quality. It builds a pipeline that converts text into traces, traces into checkable evidence, guidelines into rules, rules into code, and code outputs into process-review signals.

That is the correct direction.

Modern organizations are drowning in rules written for humans and records written by humans. The compliance fantasy is that a dashboard can sit on top and summarize reality. The operational truth is harsher: reality must first be structured, scoped, checked, and interpreted. LLMs can help with that work if they are placed inside a mechanism that produces artifacts rather than authority.

For healthcare, the paper offers a promising path toward faster guideline conformance analysis when formal guideline models are unavailable. For other regulated industries, it offers a more general design lesson: stop asking AI to declare compliance, and start using it to build the machinery through which compliance can be tested.

The machine does not need to wear the white coat. It needs to keep the audit trail clean.

Cognaptus: Automate the Present, Incubate the Future.