TL;DR for operators
How should a clinical AI system move from a noisy image to a recommendation without hiding every judgment inside one model? The practical answer is to separate image standardization, structured interpretation, retrieval, and recommendation generation so each stage can be inspected, corrected, and validated independently.
MMIR-TCM’s strongest evidence comes from removing those supports one at a time. Clinical-case memory produced the largest overall loss in prescription reasoning when removed. Formal diagnostic-theory memory mattered most for syndrome differentiation, while removing tongue findings particularly weakened prescription generation. By contrast, tongue segmentation—the architecture’s most visible component—improved semantic performance only modestly.
For clinical AI teams, the implication is operational rather than autonomous: invest in the less visible reasoning and retrieval modules, expose intermediate outputs and provenance, validate each module separately, and route poor-quality or clinically complex cases to mandatory review. The paper supports clinician-supervised documentation, case retrieval, and recommendation drafting; it does not establish safe autonomous prescribing.
The decisive evidence sits in the memories, not the mask
The operational problem is not simply whether a model can describe a tongue image. It is whether a clinician can see how the system moved from a variable-quality image to a syndrome interpretation and then to a prescription recommendation, with enough separation to identify where an error entered.
The paper’s most revealing test removes inputs from the prescription generator one at a time. Removing clinical-case memory causes the largest overall degradation. Removing formal LiuJing theory memory most strongly damages syndrome differentiation. Removing the tongue diagnosis particularly weakens prescription generation. These are ablations: their purpose is not to establish clinical efficacy, but to test whether the information sources are redundant. The observed failure patterns suggest they are not.
That finding corrects an easy misreading of the architecture. Tongue segmentation is the most visible module, but it is not the main source of reasoning performance. The system depends on different evidence channels for different decisions: prior cases supply precedent, theory supplies a structured diagnostic frame, and tongue attributes constrain the prescription context.
Three stages turn one opaque output into a reviewable workflow
The framework begins with a practical nuisance: tongue images vary in lighting, exposure, framing, and background. Its first stage, Memory-SAM, retrieves a visually similar annotated example, converts image correspondences into prompts, and uses SAM2 to isolate the tongue. This is training-free preprocessing rather than clinical reasoning. It produced clinically usable masks for 2,780 of 2,805 images, or 99.1%; 25 low-contrast or under-exposed cases required manual correction.
The second stage converts the standardized image into a fixed-format, one-sentence report covering tongue body, coating, texture, and anatomical location. A Qwen3-VL-30B model is adapted with LoRA on 2,805 expert-verified image-report pairs. This structured report is more than a narrative convenience. It is the interface between perception and retrieval: visual evidence becomes a consistent textual object that can be inspected, corrected, indexed, and passed downstream.
The third stage combines that report with patient metadata. A Qwen3-8B retrieval-augmented generator searches de-identified clinical cases and consults a separate memory derived from LiuJing theory, then generates syndrome differentiation, diagnosis, prescription, and rationale. The architecture therefore does not rely on one model to retain every visual pattern, precedent, and theoretical relation inside its parameters.
For clinical software teams, this decomposition changes the validation target. The mask, attribute report, retrieved cases, theory passages, and generated recommendation can each receive separate quality checks, provenance records, and escalation rules. A monolithic accuracy score cannot provide that control.
Masking improves consistency, but the gain is modest
The raw-versus-mask experiments are robustness and input-format ablations. Their purpose is to test whether removing background and illumination variation stabilizes report generation. The best condition—masked images during both training and inference for 10 epochs—reached BLEU-4 83.57 and TDEU 0.627. The fully raw condition reached BLEU-4 83.20 and TDEU 0.601.
The semantic increase from 0.601 to 0.627 is real within this evaluation, but it is not large enough to justify treating segmentation as the architecture’s decisive advance. Masking is better understood as a standardization control: it reduces nuisance variation, improves location scoring, and creates a cleaner artifact for human inspection. It does not replace domain adaptation, retrieval quality, or clinical reasoning.
The comparison with zero-shot public multimodal models makes a different point. Their BLEU-4 scores fall around 24–26 and their TDEU scores around 0.34–0.35, far below the domain-adapted configurations. This supports specialized adaptation on the MedTCM task. It does not show that the system is clinically effective, because both training and evaluation remain inside the paper’s retrospective domain.
The LoRA experiments serve as mechanism and sensitivity tests. At three epochs, adapting the vision tower and projector outperforms language-only adaptation, while adapting all components performs best. Higher rank and longer training improve results, with diminishing TDEU returns by 20 epochs in the reported ViTCM-LLM sequence. The practical reading is specific: specialized visual representations need adaptation; adding more language-side capacity alone does not compensate for weak visual grounding.
TDEU scores attributes, not patient outcomes
Ordinary word-overlap metrics create a measurement problem in clinical reporting. Two descriptions can use different terms for similar findings, while a fluent sentence can also omit an important attribute and still score reasonably well. The paper introduces TDEU to evaluate semantic agreement at the attribute level.
TDEU parses reports into TONGUE, COAT, LOCATION, and OTHER categories. It applies expert-curated semantic similarities, aligns predicted and reference tokens through optimal one-to-one matching, gives partial credit for related descriptions, and weights categories by clinical importance:
This is a meaningful measurement contribution because it makes clinically relevant omissions and near-equivalences more visible than BLEU alone. Its boundary is equally precise. TDEU is a TCM tongue-report metric built from configured categories, expert-curated relations, and fixed weights. It is not a universal medical-generation score, and a higher TDEU value does not establish diagnostic safety or improved patient outcomes.
MedTCM supplies scale, but only a small fraction is multimodal
MedTCM contains 124,593 de-identified records collected from four Chinese TCM hospitals between April 2021 and June 2025. The corpus includes 855 syndrome-differentiation types and 7,294 tongue-diagnosis types. Its scale matters for retrieval because similar-case search becomes more plausible when the memory contains substantial diagnostic and prescription variation.
However, the corpus should not be described as 124,593 tongue images. The curated multimodal subset contains 2,805 image-report pairs with complete records and expert verification. Prescription-generation evaluation uses 1,151 held-out cases. That distinction affects data strategy: institutions seeking a comparable system need broad longitudinal records for retrieval, but also a smaller, carefully verified paired dataset for visual-report adaptation and evaluation.
Product value comes from inspectability and routing
The paper directly shows comparative performance within a retrospective dataset, task-specific component dependence, aggregate clinician preference over GPT-4o in a limited blinded study, and recurring expert-identified errors. Cognaptus infers a supervised product pattern from that evidence: use the system to draft structured tongue findings, retrieve comparable cases, surface theory support, and prepare a reviewable recommendation for a qualified clinician.
| Product stage | Operator decision | Evidence to expose | Mandatory review condition |
|---|---|---|---|
| Image intake | Is the image usable? | Original image, mask, extraction status | Under-exposure, low contrast, failed or corrected mask |
| Report drafting | Are the attributes complete? | Tongue body, coating, texture, location fields | Missing or internally inconsistent attributes |
| Retrieval | Are the precedents relevant? | Similar cases, source metadata, theory passages | Weak similarity, sparse history, incomplete metadata |
| Recommendation | Is the synthesis clinically coherent? | Syndrome, diagnosis, prescription, rationale, provenance | Complex comorbidity, multi-syndrome pattern, clinician disagreement |
The affected users are clinicians and clinical-operations teams. The decision is whether to accept, revise, or reject a draft recommendation. The condition is an institution with sufficiently complete records, controlled image capture, and professional review capacity. The boundary is that the system assists documentation and evidence access; it does not inherit prescribing authority from benchmark performance.
The evidence supports assistance, not autonomous prescribing
The clinician study is encouraging but narrow: 12 experienced clinicians evaluated 20 cases, producing 240 ratings. Aggregate scores favored MMIR-TCM over GPT-4o, although individual cases could favor GPT-4o. This is a human-preference signal, not prospective clinical validation.
The expert error analysis defines the more consequential deployment boundary. Among 50 representative error cases, 38 contained incomplete meridian interpretation and 26 contained incomplete prescription information. These failures point to weak integration in complex or comorbid patterns—the situations where confident synthesis can be more dangerous than an incomplete draft. Performance also remains sensitive to image quality and metadata completeness, and the current framework covers tongue images plus structured metadata rather than the full set of TCM diagnostic modalities.
A deployment decision therefore requires prospective evaluation, site-specific calibration, override monitoring, and explicit abstention rules. Before any autonomous use, developers would need evidence that the system recognizes missing information, handles distribution shifts across hospitals, and improves clinically meaningful outcomes without increasing unsafe recommendations. This paper does not provide that evidence.
Build the review path before expanding the recommendation path
MMIR-TCM’s strongest contribution is architectural discipline. It shows how visual standardization, attribute reporting, case retrieval, theory retrieval, and recommendation generation can remain distinct enough to inspect. Its experiments also show why engineering attention should not follow visual prominence: the mask improves consistency, while memory removal produces the larger task-specific reasoning losses.
For operators, the next step is not to hide these stages behind a single conversational interface. It is to validate each stage, preserve provenance, route low-quality and complex cases to mandatory review, and measure clinician corrections over time. Under those conditions, the system could reduce documentation and search burden. Outside them, strong internal scores remain evidence of benchmark performance, not permission to prescribe.
Cognaptus: Automate the Present, Incubate the Future.