A Better ICD Shortlist Is Not Yet a Better Coding Workflow

TL;DR for operators

Hospital coding teams must identify appropriate diagnosis codes from clinical notes, laboratory results, and admission records. The first automation decision is whether software should predict codes directly or retrieve a shortlist that a coding professional can examine.

Bohra and Bohra’s Multimodal Dual-Encoder Retrieval for Automated ICD Coding¹ provides evidence for the retrieval approach. On MIMIC-III, its model achieves 66.75% Precision@3 versus 62.12% for the strongest multimodal classification baseline, and 62.44% versus 59.73% Micro-F1@8. A separate LLM stage improves the ordering of retrieved candidates, raising NDCG@8 from 65% to 71% on 2,000 human-annotated cases.

For hospital operations, this supports investigating retrieval-based coding assistance with optional LLM review. It does not establish that coding becomes faster, cheaper, or more accurate in production. Those outcomes require separate evaluation.

Finding codes and predicting codes are different system designs

A hospital admission can have several valid ICD diagnosis codes. A conventional multi-label classifier processes the patient record and produces a prediction score for each code in its output vocabulary.

This is straightforward to evaluate, but it ties the prediction architecture to a defined label set. Expanding or changing that set can introduce model-maintenance challenges, particularly when some codes have limited training examples.

The alternative investigated by Bohra and Bohra is to represent patient admissions and ICD codes in a shared numerical space, then search for the codes most similar to a patient record.

Their dual-encoder architecture has two parts.

The patient encoder processes clinical narratives using Clinical-Longformer, aggregating representations of text chunks. A separate multilayer perceptron processes structured information, including laboratory measurements, demographics, and admission features. A learned sigmoid gate combines the two modalities dimension by dimension into a 768-dimensional representation.

The ICD encoder uses BioClinicalBERT to represent code descriptions, categories, and hierarchy paths in the same embedding space.

Training uses a multi-positive contrastive objective: all codes assigned to an admission count as positive matches, while other codes, including clinically similar hard negatives, help the system learn distinctions.

After training, ICD embeddings can be precomputed and indexed using FAISS. The system encodes an incoming patient record and searches that index for nearby codes.

This design separates patient processing from code-catalog retrieval. It also provides a way to use descriptive code metadata during matching, rather than relying exclusively on a code-specific classification output.

The architectural argument is plausible. Whether this separation reduces latency, maintenance costs, or infrastructure requirements is a different question.

The retrieval advantage is visible across the benchmark

The authors evaluate 53,423 ICU-related hospital admissions from MIMIC-III v1.4, using a patient-level split of 70% training, 15% validation, and 15% testing.

They compare structured-only classification, text-only classification, multimodal classification, and multimodal retrieval.

An admission may have multiple correct codes. Precision@3 measures the proportion of correct codes among the first three predictions. Micro-F1@8 combines precision and recall across predictions using a fixed shortlist of eight codes, aggregating contributions across the evaluation set.

The main benchmark comparison, reported in Section V-A, Table I, is:

Two findings matter.

First, multimodal classification outperforms both single-modality baselines across all reported metrics. Structured measurements and clinical narratives contain complementary predictive information in this benchmark.

Second, changing from multimodal classification to multimodal retrieval produces another improvement. Relative to Dual Fusion, the retriever gains 4.63 percentage points in Precision@3 and 2.71 points in Micro-F1@8.

The larger improvement at the top of the shortlist is particularly relevant to coding interfaces that show reviewers only a few initial recommendations.

However, this comparison is main benchmark evidence, not a component ablation. The models differ in architecture and training objectives. The experiment does not determine whether gated fusion, contrastive learning, hard-negative selection, or code metadata accounts for a particular share of the gain.

The reported results establish a comparative advantage for the complete retriever under the evaluated conditions.

The LLM improves ranking, not the reported Micro-F1

Retrieval produces candidates through numerical similarity. But two clinically related codes can have similar embeddings while differing in the details that determine their suitability for an admission.

The proposed second stage uses Claude 3.7 Sonnet. Given a patient summary and metadata for the retrieved codes, the LLM produces relevance scores and short natural-language explanations. Its scores are then used to reorder the shortlist.

The authors evaluate this stage separately in Section V-B using 2,000 human-annotated cases with three levels of clinical relevance.

They report NDCG@8 rising from 65% to 71%, an improvement of six percentage points.

NDCG@8 measures ranked relevance: it rewards placing highly relevant candidates earlier among the first eight results. It is not equivalent to classification accuracy or Micro-F1.

This distinction changes how the result should be interpreted. The retriever’s 62.44% Micro-F1@8 was measured before the optional LLM stage. The 71% NDCG@8 result measures the subsequent ordering of candidates on a different evaluation subset.

Re-ranking also cannot recover a correct code omitted from the retrieved candidate set. Its potential contribution is better prioritization within the available shortlist.

Nor does an explanation generated alongside a relevance score prove that the explanation accurately reflects the clinical record. The study provides no quantitative explanation-faithfulness evaluation.

When should a coding system invoke the LLM?

For Cognaptus, the most actionable inference is architectural: candidate discovery and expensive contextual reasoning need not be invoked together for every case.

A hospital coding team could test retrieval as the default recommendation engine, keeping final code assignment under appropriate human review. An EHR product team could introduce LLM re-ranking only when retrieval scores are inconclusive or candidate codes require closer clinical discrimination.

That selective design is an operational hypothesis, not an experimentally established cost-saving strategy. The paper does not identify an optimal ambiguity threshold or measure the quality-cost tradeoff of selective invocation.

A practical evaluation would separate four decisions:

These measurements answer different procurement and governance questions. A system with better offline retrieval metrics may still be unattractive if re-ranking introduces excessive latency, unsupported diagnoses, or additional verification work.

Similarly, improved ranking could be valuable even without a large increase in aggregate code-prediction scores, if coding professionals can locate defensible recommendations sooner. That benefit must be demonstrated through actual reviewer behavior rather than inferred from NDCG alone.

What remains unresolved before clinical deployment

The paper’s evaluation covers ICD-9 codes and a single-center critical-care dataset. Performance on ICD-10, other institutions, different documentation practices, and broader patient populations remains unknown.

The reported experiment also lacks confidence intervals, significance testing, and repeated-seed estimates. There is no detailed component-level ablation or dedicated analysis of performance on rare codes, despite the architectural motivation around large label spaces.

The human-annotated ranking subset provides a separate relevance comparison, but its sampling description is not fully resolved in the source record, and numerical inter-annotator agreement is not reported.

Most consequentially, the paper does not quantify retrieval throughput, end-to-end latency, LLM inference costs, explanation faithfulness, or prospective coding outcomes.

These are not minor implementation details for a healthcare buyer. They determine whether the measured improvement in candidate quality translates into fewer coding errors or lower review effort.

A stronger shortlist is a starting point

Bohra and Bohra demonstrate that multimodal retrieval can outperform the evaluated classification alternatives for ICD-9 prediction, and that an LLM can separately improve candidate ordering.

The case for replacing routine classification with retrieval is therefore stronger than the case for deploying the full system autonomously.

For healthcare organizations, the next meaningful experiment is a controlled human-review workflow: measure what the retriever finds, what the LLM changes, what reviewers accept or correct, and what each stage costs.

Better candidate selection is measurable today. Better coding operations remain to be demonstrated.

Cognaptus: Automate the Present, Incubate the Future.