The Best AI Team Knows When to Stay Quiet: GRADE and the Economics of Selective Reasoning

TL;DR for operators

GRADE treats a collection of language models less like a brainstorming circle and more like an operations team with an unusually strict meeting policy.

For each query, the system learns:

• how far the request should travel through the hierarchy;
• which expert agents should be activated;
• which agents should be allowed to read one another’s work;
• which branches should be discarded before the final answer is assembled.

That restraint is the paper’s most important result. The winning configuration is not the one that activates every model and encourages maximum communication. Fixed three-agent routing beats fixed five-agent routing. Allowing every agent pair to communicate reduces MMLUPro accuracy by 2.1 points relative to the learned communication setting. Easy questions can bypass the expert pool entirely.

In the reported experiments, GRADE reaches 94.8% on GSM8K, 78.2% on MMLUPro, and 57.4% on GPQA, beating the strongest comparison system on those benchmarks while using about 17B active parameters per query instead of 28B. It loses on AIME-2025, where stronger individual reasoners still matter more than elegant orchestration.

For businesses, the relevant idea is not “deploy six agents.” It is to turn a heterogeneous model portfolio into adaptive infrastructure: cheap paths for routine work, specialist escalation for difficult cases, controlled information exchange, and model replacement without retraining the entire orchestration layer.

The uncertainty is equally concrete. The results come from four closed-form reasoning benchmarks, six nominal agents backed by three distinct frozen models, three training seeds, 200 optimization steps, and one NVIDIA H200 environment. The paper does not test real API pricing, production concurrency, tools, retrieval, long-running tasks, service failures, or open-ended enterprise workflows.

Meetings are expensive, even when the attendees are software

Most organizations eventually discover that adding people to a meeting does not guarantee better decisions. It does guarantee more calendars, more repetition, and a larger opportunity for someone to introduce a confidently irrelevant opinion.

Multi-agent AI systems have been recreating this management lesson at machine speed.

A standard ensemble may send every query to every model, collect all outputs, and then vote or synthesize. Debate systems add rounds of response and criticism. Layered agent systems pass outputs repeatedly through fixed graphs. The assumption is usually that more independent reasoning and more communication should improve the final answer.

Sometimes it does. It also multiplies token use, latency, resident memory, and opportunities for weak agents to contaminate stronger work.

Sudipto Ghosh and Tanmoy Chakraborty’s GRADE framework starts from a more operational question: before asking several models to collaborate, can the system learn whether collaboration is useful at all?¹

GRADE—Gated Routing and Adaptive Depth for Efficient Reasoning—coordinates a hierarchical pool of agents through four learned gates. Those gates decide who works, how deep the query travels, who may communicate, and which branches survive. The authors train the entire coordination policy jointly with a reinforcement-learning method called CoGRPO.

The resulting system is interesting not because it creates another committee of language models. We have several of those already, each with its own charming way of consuming a GPU budget. It is interesting because it tries to govern the committee.

GRADE’s real product is four different ways to say no

The architecture has three levels.

At level 0, a small Qwen2.5-0.5B orchestrator receives the query. It can answer directly or send the request deeper.

At level 1, two manager modules group related specialists. These managers are small trainable multilayer perceptrons, not language models generating their own prose. Their role is to bias routing toward coherent groups of experts.

At level 2, six sub-agents perform the substantive reasoning. The six agents are backed by three distinct frozen models:

• three agents share one Qwen2.5-7B-Instruct model;
• two share one Phi-3-mini-4K-Instruct model;
• one uses Llama-3.2-3B-Instruct.

That distinction matters. “Six agents” sounds like six separately resident foundation models. In this implementation, multiple agents can reference the same in-memory backend. The system’s diversity comes partly from separate adapters, descriptors, answer heads, and positions in the topology—not only from six independent model weights.

The four gates then regulate the process.

These gates are small networks. The expensive part remains the language-model inference they allow or prevent.

The Depth Gate controls escalation

The Depth Gate chooses among three paths.

A depth-0 query stops at the 0.5B orchestrator. A depth-1 query involves the manager layer. A depth-2 query reaches the full expert tier.

This is not merely a latency trick. Unnecessary depth can lower accuracy by introducing additional weak or irrelevant outputs. The system therefore applies a penalty for descending farther than needed, but keeps correctness sufficiently dominant in the reward so that the gates do not learn the wonderfully economical policy of answering everything badly with the smallest model.

The measured allocation reflects benchmark difficulty. GSM8K queries recruit an average of 2.1 agents at a mean depth of 1.8. AIME-2025 recruits 3.8 agents and always reaches mean depth 2.0.

The mechanism is therefore doing more than classifying tasks by subject. It is learning a rough estimate of how much reasoning infrastructure each query deserves.

The Assignment Gate chooses specialists rather than headcount

The Assignment Gate scores each expert using the query representation, the agent’s learned descriptor, and a shared signal from the manager responsible for that expert’s sub-pool.

The manager signal lets related experts be recruited together. A question crossing biology and chemistry, for example, may benefit from a coherent specialist group rather than the individually highest-scoring agents drawn from unrelated domains.

The gate uses independent Bernoulli decisions in the main dynamic configuration, with at least one expert guaranteed. A load-balancing loss discourages the router from collapsing onto one habitual favorite.

That regularizer matters because a nominally heterogeneous pool is not useful when one agent quietly receives every ticket.

The Cross-Read Gate limits who gets copied on the email

Once agents have produced message representations, Cross-Read decides whether agent (a) may attend to agent (b). It therefore constructs a directed communication mask rather than broadcasting every message to everyone.

The surviving messages are fused using masked multi-head cross-attention.

This separation is useful. Cross-Read decides which communication links exist; cross-attention decides how to combine the information passing through those links. The component ablations show that both contribute independently.

Communication is not free. It expands context, increases combination work, and allows an irrelevant agent to influence a useful one. GRADE explicitly treats silence as an available action.

The Prune Gate removes work that should not reach the decision

After assignment and communication, the Prune Gate gives each active branch a utility estimate. Branches below a threshold are removed, although at least one must survive.

This resembles quality control more than routing. The Assignment Gate decides who receives the task. The Prune Gate decides whose resulting work deserves consideration.

Pruning occurs before the final cross-attention combination, so it protects both answer quality and latency. In operational terms, hiring a consultant and ignoring the report is still expensive, but less expensive than circulating it through the entire executive team.

CoGRPO trains coordination without building a critic for every office decision

The four gates make discrete decisions. Ordinary backpropagation cannot pass directly through sampled agent selections, depth choices, communication links, and pruning actions.

GRADE therefore uses reinforcement learning.

Its training method, Collaborative Group-Relative Policy Optimization, samples eight coordination rollouts for each query. The rollouts vary not only in generated answers but also in depth, expert assignment, communication structure, and pruning.

Each rollout receives a reward combining four elements:

1. terminal correctness;
2. the final answer’s probability margin;
3. alignment signals for the surviving agents;
4. a token-based compute penalty.

Correctness receives a weight of 2. The normalized cost penalty can deduct at most roughly 0.12 under the reported settings. This asymmetry is deliberate: the system should economize among viable reasoning strategies, not discover that incorrect minimalism is the cheapest possible product.

The reward for rollout (i) is normalized relative to the other rollouts for the same query:

Every participating gate and surviving agent receives the same normalized advantage.

That is the “collaborative” part. CoGRPO does not attempt to calculate a separate counterfactual contribution for every gate and agent, which would require repeatedly rerunning the system with individual components disabled. Instead, it distributes the team result to all participants and relies on repeated sampling to average out noisy credit.

This is crude in the same way a shared team bonus is crude. It cannot tell whether a particular communication edge was decisive or merely present. The paper explicitly acknowledges that the signal is noisy. Its attraction is that it avoids training a separate critic over an enormous combinatorial action space.

The approach adds importance-ratio clipping, KL regularization, and within-group standard-deviation normalization. A supervised cross-entropy loss stabilizes the differentiable answer-combination components early in training, but gate decisions receive gradients only through the policy-gradient objective.

The frozen expert models themselves are not updated. Training changes the gates, lightweight adapters, answer heads, combination head, and the last two encoder layers.

The objective comparison isolates more than a new acronym

The paper compares CoGRPO with five alternative reinforcement-learning objectives while keeping the GRADE architecture and agent pool fixed. This is an objective ablation rather than a general comparison between unrelated systems.

The critic-based MAPPO and adapted MAPoRL objectives both trail the critic-free alternatives. The authors attribute this to the difficulty of fitting a value function over GRADE’s combined routing, depth, communication, and pruning space.

That explanation is plausible within this setup. The controlled comparison supports the claim that CoGRPO is better suited to this architecture than the tested alternatives. It does not prove that critics are generally inferior for multi-agent language-model coordination. A different critic, more training, different state representation, or larger workload could change the result.

Within the critic-free group, the 1.6-point gap between the MAGRPO-style objective and CoGRPO supports the additional use of standard-deviation normalization and clipped updates. It does not isolate those two features individually, because they change together.

The experiment is useful precisely because its claim is narrow. It says the training recipe matters, not that “critic-free” is now an ideology.

The headline results are strongest where routing can exploit uneven difficulty

The main comparison covers GSM8K, MMLUPro, GPQA, and AIME-2025. Scores are averaged across three training seeds. AIME results are also averaged across 16 decoding samples per problem because the benchmark contains only 30 problems.

The statistical tests use pooled two-sample (t)-tests reconstructed from three-seed means and standard deviations. The paper does not apply a multiple-comparison correction.

The practical reading is therefore more restrained than “GRADE wins three of four.”

MMLUPro is the clearest success. The 4.8-point gain is the largest, statistically significant, and aligned with the proposed mechanism: queries vary in difficulty and subject coverage, giving the system opportunities to stop early, recruit a specialist group, or share information selectively.

GSM8K also shows a statistically significant gain, but only 1.1 points. The benchmark is already comparatively easy for the 7B-class models in the pool, so there is less room for sophisticated coordination to help.

GPQA shows a larger 3.3-point numerical gain, but it is not statistically significant under the paper’s test. The result is suggestive rather than decisive. The benchmark’s domain-crossing questions fit the architecture’s manager and Cross-Read mechanisms, yet the three-seed evidence is too variable to treat the observed margin as settled.

AIME-2025 supplies the useful failure case. Puppeteer’s fixed ensemble of four Qwen2.5-7B agents reaches 27.2%, while GRADE reaches 25.3%. The authors argue that olympiad-style mathematics demands long sequential reasoning from strong individual models, and the GRADE pool includes weaker 3B and 3.8B backends. Routing cannot manufacture reasoning capacity that the selected experts do not possess.

That is an important boundary. Orchestration can allocate intelligence. It cannot create missing intelligence.

The active-compute advantage is real, but it is not the same as a cloud bill

GRADE averages roughly 17B active parameters per query. Puppeteer activates roughly 28B. On MMLUPro, GRADE also records higher accuracy while using 52GB peak VRAM compared with Puppeteer’s 56GB.

Its latency ranges from 3.1 to 11.4 seconds per query. Puppeteer takes 13 seconds under the paper’s setup.

The range is more informative than a single average. Easy queries use only the small orchestrator and finish near the lower end. Hard queries activate the specialist pool and approach the upper end.

This is the correct shape for adaptive infrastructure: inexpensive routine handling with escalation when necessary.

It is not yet a business cost model.

“Active parameters” measures the parameters evaluated across forward passes. It does not incorporate provider prices, batching, network latency, API minimum charges, cache behavior, token pricing differences, or utilization. Peak VRAM remains 52GB because the distinct expert models must be resident even when a query activates only part of the logical agent pool.

A system can therefore reduce marginal computation without proportionally reducing reserved capacity. Finance departments tend to notice this distinction eventually, usually after the architecture diagram has already been approved.

The ablations say hierarchy matters more than agent chatter

The leave-one-out component experiments retrain the system from scratch after removing each component. Their purpose is to estimate which mechanisms contribute to the full system, not to provide independent benchmark claims.

The largest drop comes from removing the hierarchy, which the experiment defines jointly as the manager level and Depth Gate. The ablation therefore does not tell us whether manager grouping or adaptive depth contributes more. It shows that the hierarchical package is central.

Cross-Read produces the next-largest loss, followed by cross-attention. These results support a distinction often blurred in agent discussions: deciding that agents may communicate and combining their messages effectively are separate technical problems.

Dynamic topology contributes only 1.1 points over a fixed-routing alternative in this leave-one-out comparison. That is useful, but modest. The paper’s broader value does not rest solely on choosing a different number of agents for every query.

The so-called persistent memory also deserves careful naming. It is an exponential moving average of combined representations updated within training batches and frozen during evaluation. It is not episodic memory, customer history, or cross-session recall. The authors describe it as a low-variance context bias.

Removing it costs 1.7 points, but no enterprise should read that result as evidence that GRADE has solved organizational memory. The mechanism does not remember last quarter’s procurement dispute, however emotionally memorable it may have been.

More agents and more communication eventually make the system worse

Two experiments directly test the paper’s central misconception.

First, the authors replace dynamic assignment with fixed (k)-agent routing.

Performance peaks at three fixed agents and then declines. Dynamic routing performs slightly better than every fixed setting.

The gain over fixed (k=3) is only 0.4 points, so the evidence does not say that dynamic routing revolutionizes accuracy. It says that adding agents beyond the useful specialist set can inject enough noise to erase the expected benefit.

Second, the Cross-Read frequency experiment varies the fraction of agent pairs allowed to communicate.

MMLUPro accuracy rises from 72.8% with no communication to 78.2% when roughly half the possible links are enabled. It then falls to 76.1% under full communication.

Selective communication therefore adds 5.4 points relative to silence, while indiscriminate communication gives back 2.1 of those points.

This is stronger evidence than simply observing that the learned gate is sparse. It shows a non-monotonic relationship: some communication is valuable, but complete communication is inferior.

The system is not merely reducing a cost while preserving accuracy. It is filtering interactions that actively hurt accuracy.

The token curves expose an unfinished AIME policy

The paper tracks tokens per query across the 200 training steps. This is a convergence diagnostic and exploratory analysis, not an additional benchmark comparison.

GSM8K token use falls from roughly 2,100 to 1,250 by step 50 and then remains fairly stable. MMLUPro and GPQA drop initially, briefly rise near step 100 as the policy re-explores routing choices, and then settle around 1,500 and 1,860 tokens respectively.

AIME-2025 behaves differently. Token use declines from roughly 4,700, bottoms near 3,100 around step 150, and rises again toward 3,500 at step 200.

The authors interpret this as incomplete convergence: the gates have not learned to prune difficult mathematical reasoning chains reliably within the training budget.

This matters when interpreting the AIME result. GRADE loses to Puppeteer there, and its own routing policy is still moving at the final checkpoint. More training might improve it, worsen it, or simply stabilize a similar score. The present evidence cannot decide.

It also matters for deployment. A routing policy trained briefly on one workload may appear economical before its compute allocation has stabilized. Cost control should be evaluated over learning dynamics, not just at a conveniently photogenic checkpoint.

Hot-swapping experts requires recalibrating the surrounding bureaucracy

Model portfolios change. A vendor raises prices. A local model improves. A compliance team prohibits a provider in one region. A specialist endpoint disappears at 4:57 p.m. on a Friday, its natural habitat.

GRADE addresses replacement through an Expert Registry and per-agent calibration maps.

Each agent’s messages pass through an affine normalization based on running statistics and learned scale and shift parameters. When one backend is replaced, the system recalibrates only that agent using 64 anchor queries. The gates and other agents remain unchanged.

On MMLUPro, preswap accuracy is 78.2%.

Without calibration, accuracy drops 18.1 points immediately and remains at 71.4 after the tenth reported evaluation batch. Calibrated replacements return close to the preswap score much faster.

The paper’s terminology is slightly untidy here. The table defines its recovery metric as the number of evaluation batches required to return within 0.5 percentage points of the original accuracy, while the prose sometimes calls these “queries.” The reliable conclusion is comparative: calibrated swaps recover within the first nine reported evaluation units; the uncalibrated condition does not recover within 20.

The mechanism is operationally important because the gates depend on model-output distributions. A replacement may produce embeddings with different scale, variance, confidence, or alignment. Even when the new model is nominally capable, thresholds learned around the old model can silently stop working.

This is the paper’s most transferable engineering lesson. Modular replacement requires interface calibration, not merely matching an API schema.

The business opportunity is governed model portfolios, not agent abundance

The paper directly shows performance and efficiency on reasoning benchmarks. Cognaptus’ business interpretation is broader, but it should remain distinguishable from the evidence.

Three practical pathways stand out.

Route by expected value, not model prestige

Many enterprises currently route by static rules: one inexpensive model for general work, one premium model for important work, and perhaps a fallback after failure.

GRADE suggests a richer policy. Difficulty, domain coverage, uncertainty, and expected collaboration value can jointly determine the path. The cheapest appropriate route may be a direct answer, a single specialist, or several specialists with controlled information exchange.

The ROI opportunity comes from avoiding premium computation where it is unnecessary while preserving escalation for genuinely difficult tasks.

The evidence does not yet provide a ready-made routing policy for invoices, contracts, support cases, or market research. It provides a mechanism worth testing on those workloads.

Treat communication as a budgeted resource

Agent frameworks often focus on which models participate and neglect who can see whose output. That omission can create large context windows filled with redundant intermediate reasoning.

A production version of Cross-Read could regulate:

• whether a legal agent reads a sales agent’s draft;
• whether a retrieval agent passes evidence to every downstream model;
• whether a numerical verifier sees the full narrative or only extracted claims;
• whether regional agents share potentially restricted data.

Selective communication can affect cost, latency, answer quality, and governance simultaneously. The paper tests only accuracy-oriented message masking, but the control point is commercially useful.

Separate model replacement from orchestration retraining

Per-agent calibration creates a narrow adaptation layer between expert outputs and shared routing logic.

That could reduce switching costs when replacing a vendor model, moving a workload on-premises, or upgrading one specialist independently. The realistic objective is not perfectly interchangeable models. It is bounded disruption.

GRADE’s 64-anchor-query procedure is promising, but the experiment covers a small set of replacements on one benchmark. A business deployment would need anchor sets representing actual traffic, safety-critical edge cases, regional variations, output schemas, and failure modes.

Calibration should also be paired with rollback rules. A model that returns to benchmark accuracy may still violate latency, formatting, refusal, or compliance requirements.

What the paper does not establish

The boundaries are not decorative footnotes. They determine what can responsibly be taken into production.

Four benchmarks are not an operating environment

GSM8K, MMLUPro, GPQA, and AIME-2025 have objectively gradable answers and relatively compact prompts. Correctness can be turned into a clear reward.

Enterprise tasks often have delayed, partial, disputed, or multi-objective outcomes. “Was this strategic analysis useful?” is less cooperative than “Which answer option is correct?”

CoGRPO’s reward design would need substantial adaptation where quality depends on factual support, policy compliance, user satisfaction, style, cost, and downstream outcomes at once.

The system does not use tools or retrieval

No experiment includes search, databases, code execution, external tools, or retrieval-augmented generation. Those operations introduce variable latency, nondeterministic failures, permissions, and side effects.

Routing models is easier than routing models that can also alter records, contact customers, or transfer money. One hopes.

The pool is heterogeneous, but narrowly so

The six agents use three distinct local model families in the main configuration. API-to-local and local-to-API swaps are tested separately, but the main benchmark evidence does not represent a large commercial registry spanning many providers, context windows, modalities, prices, and service-level agreements.

The architecture may scale. The paper does not demonstrate that scaling.

Three seeds provide limited statistical resolution

The reported standard deviations and significance tests are useful, but three training seeds leave substantial uncertainty, particularly on GPQA and AIME. The statistical comparison also omits correction across the four benchmark tests.

MMLUPro is the most convincing result. GPQA is promising but inconclusive. AIME is a documented loss with an unconverged allocation policy.

One H200 does not reveal production throughput

All experiments run on a single NVIDIA H200 NVL with 141GB of memory. The latency and VRAM figures are therefore implementation-specific.

A commercial deployment may distribute experts across devices or providers, batch requests, cache prefixes, or wait on network endpoints. Under those conditions, the optimal gate policy could differ substantially.

Training is short and workload stability is assumed

GRADE trains for 200 gradient steps. Its token curves show that the hardest task has not stabilized by the end.

The evaluation also assumes a fixed benchmark distribution. Real traffic changes. A router optimized on last quarter’s workload may overuse the wrong specialists next quarter, especially after product launches, regulatory changes, or shifts in customer behavior.

Adaptive routing creates a monitoring obligation. The orchestration policy itself becomes a model that can drift.

The best multi-agent system may look less collaborative

GRADE’s contribution is not that it persuades several models to reason together. It is that it gives the system multiple opportunities to prevent collaboration from becoming wasteful.

The Depth Gate can stop escalation. The Assignment Gate can decline unnecessary experts. Cross-Read can block irrelevant conversations. The Prune Gate can discard weak work. CoGRPO trains those decisions through a shared outcome signal, and calibration reduces the damage when one expert is replaced.

The reported gains are meaningful but uneven. MMLUPro provides the strongest evidence. GSM8K offers a smaller improvement. GPQA points in the expected direction without reaching statistical significance. AIME shows that superior routing cannot compensate indefinitely for weaker underlying reasoners.

For operators, the architecture suggests a more mature view of model portfolios. The objective is not to accumulate the largest possible cast of agents. It is to make model participation conditional, communication selective, and replacement controlled.

The fashionable multi-agent question has been: how do we make the models collaborate?

GRADE asks the more expensive question first: should they?

Cognaptus: Automate the Present, Incubate the Future.