One Harness, Many Failure Modes

TL;DR for operators

A platform team supporting several language models can achieve respectable task coverage while still delivering inconsistent results across those models. The relevant question is not only whether some supported model completes a task, but whether the same agent harness works reliably with every supported model.

MaD-RSI, proposed by Yu Cui and colleagues,¹ addresses this problem by using execution successes and failures from heterogeneous, frozen LLMs to improve one shared harness. On MedXpertQA, it matches the strongest single-model optimization baseline’s 60% four-model success coverage while raising joint success from 13% to 18%.

For multi-model agent operators, the approach offers two possibilities: better cross-model consistency and less duplicated optimization work. The cost qualification matters. Training-token savings of 18.3%–55.9% on three general-task benchmarks are measured against four independent optimization runs combined. SafetyBench instead records a 15.2% increase. These are preliminary benchmark findings, not established production economics.

The same task coverage can conceal different reliability

Consider an AI platform offering the same agent workflow through four LLM backbones. Its engineering team can optimize separate instructions and execution policies for each model, or maintain a shared harness intended to work across all four.

Suppose two harnesses allow at least one model to complete 60% of test cases. On that measure, they appear equivalent. Yet one might allow all four models to succeed on only 13% of cases, while the other achieves 18%.

That is precisely the distinction observed on MedXpertQA in the paper’s four-model evaluation (Section 4.2, Table 1).

The authors formalize these two outcomes:

• Pass@M measures the proportion of aligned query-trial observations on which at least one of the evaluated models succeeds.
• Pass^M measures the proportion on which every evaluated model succeeds.

Here, M counts distinct models, not repeated generations from one model. Pass@M is an oracle-style coverage measure; it does not establish that a deployed system can identify the successful model in advance.

For a platform that routes work among interchangeable providers, joint success captures a different engineering requirement: a task should remain solvable when the assigned backbone changes. Neither metric replaces individual-model reliability, but treating them as interchangeable can conceal a material portability problem.

Model disagreement becomes optimization evidence

Conventional single-model recursive self-improvement uses execution feedback from one LLM to revise its agent harness. Other models may subsequently be tested for transfer, but their distinctive failures did not inform the original optimization.

MaD-RSI changes where the evidence comes from.

Four frozen development models—DeepSeek-V4-Flash, DeepSeek-V3.2, Qwen3.8-Flash, and Qwen3.8-27B—execute common queries under matched sampling configurations. Their observed success rates are organized by query, model, and sampling configuration, with associated execution trajectories retained for diagnosis.

The method then selects diagnostic queries using two complementary signals.

Cross-model disagreement identifies queries on which models exhibit different empirical success rates. An entropy-based score compares uncertainty in the weighted aggregate success rate with the weighted average uncertainty of individual models. Calculations are performed within sampling configurations before aggregation.

Response-pattern coverage favors diagnostic examples exhibiting distinct combinations of model outcomes. The selection procedure also preserves unanimous-success and unanimous-failure examples as anchors when available. These cases can reveal shared capabilities or deficiencies even though their disagreement score is zero.

The experiments use temperatures of 0.0 and 1.0, two repetitions per query-model-temperature combination, and equal aggregation weights across development models.

Selected trajectories are subsequently examined together. A successful execution can provide a contrast for understanding a model-specific failure, while failures shared across models may suggest a deficiency in the common harness. Diagnostic claims must reference the underlying execution evidence before an LLM proposes revisions.

General-task experiments run three optimization rounds with two candidate slots per round.

The resulting harness is intended to operate with each model independently. MaD-RSI neither fine-tunes the underlying models nor combines their predictions through an inference-time ensemble.

This is a meaningful shift in development practice: model diversity supplies information for improving a common execution process, rather than serving only as a final portability test.

Joint success improves more consistently than coverage

The paper’s principal comparison evaluates the shared harness against the original harness and four independently optimized alternatives. RRSI supplies the single-model optimization baseline for general tasks; SHE supplies the safety baseline.

Selected four-model results are:

Source: Section 4.2, Table 1. Baseline values are the best reported single-model-optimized results for the respective metric and need not come from the same harness.

LawBench provides the clearest coverage improvement. MaD-RSI reaches 73.00% Pass@4, exceeding the strongest independently optimized RRSI result by 11.50 percentage points. Joint success increases by 4.50 points over the strongest RRSI counterpart.

MBPP+ and MedXpertQA show a more revealing pattern: coverage matches the strongest baselines, while joint success improves by 1.50 and 5.00 percentage points, respectively.

SafetyBench extends the comparison to attack safety, utility under attack, and clean safety. MaD-RSI obtains the highest reported four-model joint-success result on all three measures, including a 7.27-point improvement in utility under attack over the strongest SHE baseline.

The main finding is therefore not uniform improvement across every outcome. It is a more consistent improvement in simultaneous success across the tested model population.

Two additional models provide a limited portability test

The authors then add Doubao-Seed-2.0-Mini and GLM-5.3-Flash, neither of which contributed evidence during shared-harness optimization.

This expanded evaluation is a transfer robustness test, rather than a separate optimization experiment or component ablation.

MaD-RSI exceeds the preselected single-model baselines in Pass^6 across all six reported dataset-metric combinations. MedXpertQA joint success rises from 8.50% to 13.00%, while SafetyBench utility-under-attack joint success rises from 66.36% to 76.36% (Section 4.2, Table 2).

Coverage still requires separate inspection. On SafetyBench utility under attack, Pass@6 falls slightly from the baseline’s 96.36% to MaD-RSI’s 95.45%.

These results indicate transfer to the two additional evaluated models under the reported benchmarks. They do not establish portability to arbitrary future architectures or guarantee that every success metric will improve.

Development savings depend on the comparator

A shared harness could avoid repeatedly optimizing similar workflows for each supported backbone. Whether this is economical depends on how much optimization would otherwise be duplicated.

Against the combined training-token consumption of four independently optimized harnesses, MaD-RSI reports:

Source: Section 4.2, Figure 2. Negative values indicate reduced consumption relative to the combined four-run baseline.

The study also reports lower inference-token consumption and inference execution time across all four benchmarks relative to the aggregate expenditure of the independently optimized baselines. These resource comparisons do not establish lower production-serving latency.

The authors formalize the choice using a performance score less a cost-weighted development expenditure. Shared optimization is preferable under that criterion when its performance disadvantage, if any, is no greater than the value assigned to its cumulative cost savings.

For an infrastructure team, this makes the comparison conditional on the number of models requiring optimization, the workload, the chosen performance objective, and the value placed on development resources.

A team supporting only one backbone cannot claim the reported savings against four independent runs. A team supporting several interchangeable backbones has a more relevant comparison, but should still measure its own development costs.

What remains unverified before deployment

The authors explicitly describe the work as preliminary and ongoing.

Three boundaries matter for deployment decisions.

First, the experiments evaluate an integrated method without reported component-isolating ablations. The separate effects of disagreement scoring, pattern coverage, anchor preservation, and joint diagnosis therefore remain unresolved.

Second, the comparisons provide no reported confidence intervals or significance tests. They establish observed benchmark differences under the evaluated protocol, not statistically quantified guarantees.

Third, the study does not measure long-term harness maintenance, changing provider versions, production reliability, or integration costs. Disagreement identifies potentially informative cases, but does not itself determine why an execution failed.

For a multi-model platform, Cognaptus therefore sees the appropriate next step as a controlled internal comparison: optimize a shared harness using aligned cross-model trajectories, compare it with separately optimized configurations, and measure individual accuracy, joint success, attack-related outcomes, and total development expenditure on representative workloads.

Treat model diversity as evidence before treating it as a deployment feature

MaD-RSI’s most consequential contribution is its treatment of heterogeneous model behavior as evidence for harness development. Instead of optimizing against one model and discovering portability failures afterward, the approach incorporates those differences into the revision process.

Its benchmark results make a credible preliminary case for improved cross-model consistency. Its resource measurements show why that consistency may sometimes be economical, while the safety benchmark demonstrates that shared optimization is not automatically cheaper.

For organizations maintaining agents across several LLM providers, the decision is whether the observed improvements in shared reliability justify the actual development costs under their workloads. MaD-RSI supplies a testable methodology for making that decision, rather than a universal answer.

Cognaptus: Automate the Present, Incubate the Future.