Too Many Doctors in the Room? Benchmarking the Rise of Medical AI Agent Teams

Doctors know the problem.

A difficult case enters the room. One specialist sees a radiology pattern. Another notices a metabolic clue. A third worries about a rare diagnosis. Everyone has a useful fragment. Then the meeting gets longer, the notes get messier, and somehow the final answer becomes less clear than the first opinion.

This is roughly where medical AI agent systems are heading. The current fashion is to stop asking one model for an answer and instead assemble a small committee of models: one agent reasons, another debates, another audits, another pretends to be a radiologist, and a final agent synthesizes the conclusion. It sounds reassuring. It also sounds expensive. And, if nobody manages the meeting, it can become a very sophisticated way to waste tokens.

The paper behind MedMASLab is useful because it does not merely introduce another medical agent team. It introduces a framework for comparing them under shared conditions.¹ That distinction matters. In medical AI, the question is no longer simply whether a model can answer a clinical question. The harder question is whether different orchestration designs can be compared fairly across modalities, diseases, backbones, costs, and failure modes.

MedMASLab’s answer is not “multi-agent systems work.” That would be too easy, and also not quite true. Its answer is closer to this:

Medical AI agent teams sometimes help, but their gains depend on the task, the backbone model, the communication pattern, the evaluation method, and whether the agents can still follow instructions after several rounds of clinical theatre.

A small detail. Naturally, the small detail is the whole business.

The real contribution is a comparison machine, not another AI doctor

Most medical agent papers have a structural problem. They propose a workflow, test it on selected tasks, and report improvement. That is useful, but only locally. It does not tell us whether the same agent design survives when the image modality changes, when the benchmark changes, when the backbone model changes, or when the evaluation metric stops being friendly.

MedMASLab targets that infrastructure gap.

The framework standardizes medical multi-agent system evaluation across 11 integrated MAS methods, 11 clinical benchmarks, 24 medical modalities, 11 organ systems, and 473 diseases. The included datasets cover text-only medical QA, visual question answering, video QA, diagnostic decision-making, and reasoning-chain evaluation. This is not a narrow “we tried a chatbot on radiology” experiment. It is closer to a test bench for medical agent orchestration.

The key technical move is separation. MedMASLab decouples:

This is why the paper is best read as a comparison study, not a model paper. The authors are not asking us to fall in love with one agent architecture. They are asking us to stop comparing medical agent systems with experimental setups that quietly change half the conditions.

That is the unglamorous part of AI progress. The leaderboard is only meaningful after the measuring tape stops stretching.

Rule-based evaluation was grading punctuation, not medicine

The paper’s second major contribution is its attack on evaluation brittleness.

In many multiple-choice medical benchmarks, the evaluation pipeline wants a clean output such as “B”. Real agent systems often produce long reasoning traces, arguments among specialists, intermediate hypotheses, and final explanations. The answer may be clinically correct, but the format may not satisfy a rigid parser. In ordinary benchmarks, this is annoying. In medical AI, it is dangerous because it hides the difference between a reasoning failure and a formatting failure.

MedMASLab compares five evaluation protocols:

The evidence is blunt. On PubMedQA, MDTeamGPT ranks first under the semantic judge with 79.40%, but drops to last place under the multi-regex rule with 0.40%. DyLAN reaches 71.60% under the semantic judge but falls to 0% under exact match. Other methods swing sharply depending on whether the evaluator can extract the final answer from a verbose multi-agent discussion.

This does not mean semantic judging is perfect. It means the old rules were often evaluating obedience to output style rather than clinical reasoning. That distinction is not cosmetic. A medical AI system that reaches a correct conclusion but fails JSON formatting is an engineering problem. A system that produces a fluent but wrong diagnosis is a clinical reasoning problem. Treating both as the same failure makes the benchmark less informative.

MedMASLab’s VLM-based semantic judge also receives the same multimodal context as the agent system. That matters for medical images and videos. A text-only judge can decide whether an answer sounds plausible; it cannot verify whether the answer is grounded in the radiograph, pathology image, or video frame. The paper’s evaluation design therefore moves closer to the real question: not merely “did the model say the expected words?” but “does the answer match the clinical evidence?”

For business readers, this is the first practical lesson. Before buying or building medical AI agents, inspect the evaluation layer. A beautiful agent workflow evaluated by a brittle parser is not a trustworthy system. It is a theatre performance with a broken scoreboard.

More agents do not automatically create better medicine

The obvious belief is that more agents should improve accuracy. More specialists, more debate, more checks, better answer. Lovely. Also not what the evidence says.

MedMASLab’s compute-scaling experiments compare configurable frameworks such as DyLAN, Debate, MedAgents, and MDTeamGPT on MedQA and MedVidQA. The result is not a simple upward curve. Increasing the number of agents can help up to a point, but after that point, performance may flatten or even decline while token cost keeps rising.

One concrete example: on MedQA, MDTeamGPT reaches its best performance when the agent count is set to 8. Adding more agents reduces performance. Debate also has task-specific optima: the best configuration on MedQA differs from the best configuration on MedVidQA.

This is not surprising if we stop romanticizing “collaboration.” A multi-agent system is not a panel of wise doctors. It is a communication protocol wrapped around stochastic models. Each added agent creates another opportunity for useful critique, but also another opportunity for drift, redundancy, contradiction, formatting failure, and premature consensus.

The paper’s token-efficiency analysis makes the trade-off clearer. Token use works as a proxy for communication density. In tasks requiring high-order synthesis, such as diagnostic decision-making in DxBench, more deliberation can support better reasoning. In retrieval-heavy or highly specialized tasks such as MedXpertQA, extra exchanges can introduce noise and diminishing returns.

That gives us a cleaner rule:

Agent communication has value only when the task benefits from decomposition more than it suffers from coordination overhead.

This is the part many agent demos skip. They show the conversation, not the opportunity cost of the conversation.

The backbone model can make the same architecture behave like a different product

One of the strongest findings in the paper is that “collaborative gain” is not an intrinsic property of an agent architecture. It depends heavily on the backbone model.

MedMASLab tests MAS methods with Qwen2.5-VL-7B, LLaVA-v1.6-7B, and GPT-4o-mini. The same orchestration pattern can look stable with one backbone and unstable with another. In most benchmarks, changing the backbone reorders method rankings, except in a few cases such as DxBench and M3CoTBench where rankings are more stable.

The most striking example is MDAgents on MedQA with LLaVA-v1.6-7B, where token consumption escalates to roughly 150,000 tokens per query, nearly 100 times higher than with other backbones. The paper interprets this as a termination and consensus problem: weaker instruction-following prevents the system from closing the loop, so the agents keep talking.

This is an operationally important result. Many companies treat orchestration as reusable middleware: plug in a different model, keep the agent workflow, expect similar behavior. MedMASLab suggests that this assumption is risky. The workflow and backbone co-produce the system. Change the model and the same “agent product” may become more accurate, less accurate, cheaper, slower, or completely unstable.

For procurement and deployment, that means an agent architecture cannot be certified in isolation. It must be evaluated with the exact model family, model size, context configuration, prompt policy, output schema, and inference limits that will be used in production. Otherwise, the benchmark is testing a cousin of the product, not the product.

Bigger models help, but they also change the reason to use agents

The paper’s model-scaling experiments add another complication. As Qwen2.5-VL model size increases, both single-agent and multi-agent systems generally improve. But the marginal value of multi-agent collaboration is not constant.

On MedQA, the largest collaboration gains appear with the 32B model. On MedXpertQA-MM, the largest gains appear with the 7B model. Appendix results further complicate the neat “bigger is always better” story: on PubMedQA, at the 72B scale, most MAS methods and the single-agent model perform worse than their 32B counterparts, with MedAgents as an exception.

The interpretation is not that scaling is useless. The interpretation is that scaling changes the bottleneck.

For smaller models, agents may compensate for weak reasoning by forcing decomposition, debate, or consensus. For stronger models, the single model may already perform much of the reasoning internally, so external orchestration adds less. In some cases, orchestration may amplify improvements. In other cases, it may amplify degradation.

This is a useful correction to the “AI hospital team” narrative. The question is not whether agents are better than single models. The question is when the external coordination layer adds value beyond what the base model already does.

A business translation:

The unpleasant but practical answer is that agent teams are not a universal upgrade. They are a design choice that must earn its cost.

Role-playing doctors can be expensive role-play

Medical MAS papers often assign agents specialist identities: cardiologist, radiologist, surgeon, pathologist, senior clinician. This feels intuitive because real medicine uses specialists. The danger is that role simulation can become a comforting metaphor rather than a performance mechanism.

MedMASLab directly tests medical expert-playing modes: no expert role-play, fixed expert roles, and dynamic expert role assignment. The tests use frameworks such as MDTeamGPT, Debate, and MedAgents across PubMedQA, MedQA, MedVidQA, DxBench, and M3CoTBench.

The finding is wonderfully inconvenient. Fixed and dynamic expert-playing can substantially increase token cost, sometimes explosively, without reliably improving performance. In Debate, fixed or dynamic expert modes slightly improve M3CoTBench performance but reduce MedVidQA performance, while token cost reaches an average of around 50,000 tokens per query.

So the business question should not be “does the system simulate a multidisciplinary team?” It should be:

1. Which role receives which evidence?
2. What information can each role add that the base model would not already produce?
3. How is disagreement resolved?
4. How is visual grounding checked?
5. What is the cost per incremental correct answer?

Without those answers, “medical expert agents” may simply mean longer prompts wearing white coats.

The failure modes are partly clinical, partly bureaucratic

The error analysis is especially valuable because it separates reasoning failures from system failures.

For Reconcile, the framework depends on extracting candidate answers across rounds for voting. That means the model must follow a strict format, often JSON-like. With Qwen2.5VL-3B, the system suffers severe formatting errors on complex text-heavy tasks: 84.00% on PubMedQA and 75.33% on MedQA. With Qwen2.5VL-7B, these formatting errors fall to 0.00% on PubMedQA, MedVidQA, and SLAKE-En, and to 0.55% on MedQA.

This is not a minor implementation note. It shows that the difference between a failing and functioning multi-agent system can be whether the model can reliably fill the form after a long conversation. Bureaucracy defeats intelligence. Very healthcare, really.

The MDTeamGPT error analysis with LLaVA-v1.6-7B on MedQA shows another split. Failure samples account for 58.2% of all samples. Of those, 41.9% come from incorrect model responses, while 14.0% come from hitting the round limit. Appendix examples include JSON parsing failure, wrong diagnosis, max-round termination, and cases where the model claims there is no answer.

These are different problems requiring different fixes.

This is where MedMASLab’s engineering value becomes clear. It does not merely say that a method is right or wrong. It helps identify whether failure comes from medical reasoning, communication protocol, formatting discipline, visual grounding, or cost management.

For real deployment, that diagnosis of failure is often more valuable than one more accuracy number.

What this means for healthcare AI teams

The business relevance of MedMASLab is not that hospitals should immediately deploy multi-agent diagnostic teams. Please do not turn benchmark enthusiasm into procurement policy. That road has potholes, lawyers, and patients.

The more practical implication is that healthcare AI teams need an evaluation-first workflow before agent orchestration becomes a product decision.

A reasonable deployment path would look like this:

The important word is “this.” This task. This modality. This hospital workflow. This cost envelope. This regulatory environment.

MedMASLab does not prove that multi-agent systems are clinically safe. It does not solve privacy, liability, clinician adoption, workflow integration, or regulatory validation. It also relies on a VLM-based judge, which improves over rigid matching but introduces its own trust question: who evaluates the evaluator?

Still, the paper gives teams a better way to ask the right questions. Instead of asking whether agent systems are “better,” ask where they are better, under which backbone, at what cost, with which failure modes, and according to which evaluator.

That is less exciting than a demo. It is also how real systems get built.

The appendix is not decoration; it is where the product risks show up

The appendix material should not be treated as spare parts. It clarifies how the framework becomes operational.

Appendix A details the dataset mix: MedQA, PubMedQA, MedBullets, MMLU clinical topics, SLAKE-En, VQA-RAD, MedVidQA, Med-CMR, DxBench, MedXpertQA-MM, and M3CoTBench. The point is not just breadth. The point is heterogeneity. Some tasks are text-only. Some are image-based. Some are video-based. Some test diagnosis. Some test reasoning-chain evaluation. A system that looks strong on one slice may not survive the next.

Appendix B describes how the authors adapt existing single-agent and multi-agent methods into the shared framework. This is implementation detail, but it supports the main comparison claim: the framework is not comparing one native implementation against another random codebase. It is forcing different methods into a common multimodal protocol.

Appendix C adds robustness and sensitivity evidence. Figures 11, 12, and 13 extend cost-performance, expert-playing, and model-size analyses. The model-size finding is especially useful: increasing model scale does not necessarily improve performance across all tasks and methods. That weakens any simplistic “just use a bigger model” conclusion.

Appendix D provides failure examples. This is where the paper becomes more concrete. JSON parsing failure, wrong clinical reasoning, round-limit failure, and “none of the above” errors are not abstract risks. They are the kinds of mundane breakdowns that decide whether an agent system behaves like a clinical assistant or a committee that forgot to file the report.

Appendix E describes the graphical interface: API setup, guide/documentation, quick tests, batch evaluation, and a low-code custom MAS builder. This is not the scientific core, but it matters for adoption. A benchmarking framework that only a small research group can operate is not infrastructure; it is a private lab instrument. A UI lowers the cost of repeated comparison and ablation.

The decision map: when to add doctors to the room

The main lesson of MedMASLab can be condensed into a practical decision map.

This is the article’s central correction to the industry’s agent narrative. Multi-agent systems are not magic because they resemble human teams. They are useful only when the structure of collaboration matches the structure of the task.

A multidisciplinary team works because the members bring different expertise, evidence, authority, and accountability. A multi-agent AI system must earn the same privilege. Otherwise, it is just one model talking to itself with extra billing.

Conclusion: the missing specialist is evaluation

MedMASLab arrives at a useful moment. Medical AI is moving from single-model prediction toward orchestrated reasoning systems, but the evaluation culture is still catching up. The paper shows that medical multi-agent systems can improve reasoning depth, especially when tasks require synthesis across evidence types. It also shows that these systems are fragile: rankings shift with the backbone, token costs can explode, role-play may not help, and formatting failures can masquerade as clinical failure.

The right takeaway is not cynicism. It is discipline.

The future of medical AI agents will not be decided by who creates the most elaborate simulated hospital meeting. It will be decided by who can standardize inputs, compare architectures fairly, measure semantic correctness, track cost, audit failures, and define where the system is allowed to act.

The AI doctors may be entering the consultation room. MedMASLab reminds us that someone still needs to chair the meeting, keep minutes, check the evidence, and stop the debate before it reaches 150,000 tokens.

In other words, the missing specialist is not another agent.

It is evaluation.

Cognaptus: Automate the Present, Incubate the Future.