Improve the Harness Without Forgetting What Worked

TL;DR for operators

A recurring failure in agent improvement is surprisingly mundane: a rewrite fixes one weakness and silently removes something that was already working.

PluginRSI addresses that problem by changing the unit of optimization. Instead of asking an optimizer to repeatedly rewrite the complete surrounding agent system, it represents roles, tools, skills, and memory as versioned plugins. Individual plugins can be changed and tested while most of the system stays fixed; the overall workflow is then recomposed from the growing library of successful components.

The distinction matters empirically. On held-out SWE-bench Verified tasks using Kimi-K3, the complete method resolves 69% of tasks. Removing plugin mutation drops that result to 57%; removing harness recomposition drops it to 59%. Simply giving a monolithic optimizer access to a plugin library does not reproduce the full gain.

For agent-platform teams, the practical proposition is not “build a bigger plugin marketplace.” It is to make operational mechanisms independently testable, retain successful versions, and periodically reconsider how those mechanisms work together. That can turn agent optimization from repeated rewriting into cumulative engineering. The evidence is promising but bounded: experiments cover fixed benchmarks and short optimization runs, cross-domain QA transfer is weaker, independent component tests can miss interaction effects, and the method consumes more tokens per solver rollout.

Whole-system rewrites make improvement hard to preserve

An agent’s behavior comes from more than its underlying model. The workflow determines what roles are invoked, what tools are available, how skills are loaded, what memory is retrieved, and how those pieces coordinate.

That creates a problem for recursive improvement. Suppose an optimizer rewrites the entire surrounding system after observing failures. The new version may repair the target failure, but several mechanisms change simultaneously. If performance improves, it becomes difficult to tell which change helped. If a useful mechanism disappears during the rewrite, the optimizer may later have to rediscover it.

PluginRSI, proposed by Shi and colleagues,¹ treats this as a search-design problem. The policy model and proposer model remain fixed. What evolves is the external harness through which the model operates.

The method represents a harness as a workflow plus a set of standardized, versioned plugins covering four categories: roles, tools, skills, and memory. The seed library contains 75 plugins: 20 roles, 16 tools, 31 skills, and eight memory components.

That modularity creates a controlled unit of change.

Local mutation gives individual mechanisms a cleaner test

During plugin mutation, the optimizer changes one plugin while holding the workflow and other plugins fixed. Candidate mutations are evaluated on balanced minibatches containing solved and unsolved validation tasks. Variants that improve performance can be retained in the shared library.

This is the paper’s answer to a credit-assignment problem. If one mechanism changes and the surrounding system remains comparatively stable, the observed performance difference is easier to attribute to that mechanism.

The appendix gives concrete examples. A failure-aware terminal-feedback tool improves its candidate minibatch from 10/20 resolved tasks to 14/20. A memory mechanism for incremental evidence retrieval moves from 10/19 to 14/19. Other successful mutations modify repair-scope control, semantic source preservation, precedence checking, and test-outcome classification.

These examples are implementation evidence rather than the paper’s main performance result. Their role is to show what the search process can discover when mechanisms are independently replaceable.

But local optimization alone is insufficient. Individually useful components still have to operate as one system.

Recomposition tests whether the pieces still work together

After plugin mutation, PluginRSI performs a second search step: harness recomposition. The optimizer selects components from the updated library and revises the workflow that coordinates them. A recomposed harness replaces the incumbent only when validation performance improves.

The ablation study is the strongest evidence for this two-level design.

These are ablations, not separate headline benchmarks. Their purpose is to test whether the proposed components materially contribute to the complete system.

The pattern also corrects an easy misreading of the architecture. A plugin library by itself is not the main result. Adding one to Meta-Harness moves Kimi-K3 held-out performance from 63% to 64%. Removing either of PluginRSI’s active search stages produces much larger declines.

The contribution is therefore the combination: isolate mechanisms enough to evaluate and preserve them, then periodically optimize their coordination.

The gains survive some changes in model backbone

The main SWE-bench comparison uses 400 held-in tasks for search and 100 held-out tasks for evaluation. With GPT-5.6 Terra as the optimization backbone, PluginRSI reaches 65% held-out resolve rate, compared with 55% for the strongest corresponding baseline. With Kimi-K3 optimization, it reaches 69%, compared with 63% for Meta-Harness.

The selected harnesses are then frozen and run with different solver backbones. A harness optimized with GPT-5.6 Terra reaches 60% held-out when run with Kimi-K3 and 52% with GLM-5.2. A harness optimized with Kimi-K3 reaches 64% with both GPT-5.6 Terra and GLM-5.2.

That transfer test matters because it separates harness improvements from changes to model weights. Some discovered operating mechanisms continue to help when the solver changes.

The pattern also appears outside SWE-bench. On Terminal-Bench-2.1, PluginRSI reaches 73.3% held-out resolve rate versus 66.7% for Meta-Harness.

QA shows a more restrained result. Overall accuracy rises from 0.638 for Meta-Harness to 0.675 for PluginRSI, but held-out-domain accuracy moves only from 0.568 to 0.588. The stronger evidence is therefore portability across solver backbones and related task settings, not universal transfer across substantially different domains.

A plugin library can become accumulated search memory

The library grows from 75 to 132 plugins over 15 evolution steps. That growth matters because successful mutations remain available to later candidates rather than disappearing when a new harness is proposed.

The reuse experiment tests whether this accumulated library changes future search. Starting again from the initial harness but reusing the evolved library reaches 69% held-out resolve rate after two evolution steps, compared with 65% without reuse.

For an engineering organization, this is the most consequential result to examine further. A validated role definition, debugging skill, memory routine, or tool wrapper need not remain embedded inside one historical agent configuration. With standardized interfaces and versioning, it can become an independently tracked implementation asset that later systems can reuse.

That is a Cognaptus inference from the architecture and reuse result, not evidence that production organizations will obtain the same economics. The experiment measures benchmark optimization, not organizational maintenance cost.

The practical trade-off is better reuse, not free optimization

PluginRSI reaches stronger harness performance with fewer solver rollouts than Meta-Harness, but each solver rollout is more expensive in tokens: reported input-token usage rises 7.5% and output-token usage rises 19.8%.

The resulting architecture also grows. Candidate harness implementations reach roughly 1,300 lines of code, although the coordinating workflow remains below 300 lines. Meanwhile, the authors identify management of an expanding plugin library as an open problem.

There is also a structural limitation in the mutation procedure. Holding the rest of the harness fixed makes local evaluation cleaner, but it can reject mechanisms whose benefit appears only when another plugin or the workflow changes at the same time. Modular credit assignment reduces one kind of entanglement; it does not eliminate interaction effects.

Finally, the experiments cover fixed task sets and a limited number of evolution iterations. SWE-bench held-out results measure generalization across held-out tasks within the benchmark, not transfer to entirely unseen repositories. Long-running evolution under changing production distributions remains untested.

Agent improvement becomes an asset-management problem

PluginRSI’s strongest contribution is not that plugins are modular. Modular agent components already exist. The paper shows a way to use modularity as the structure of the optimization process itself.

A production analogue would require three disciplines: mechanisms must have standardized interfaces, candidate changes must be evaluated independently enough to support credible comparisons, and successful versions must remain available for later recombination. Global workflow evaluation is then needed because component quality does not guarantee system quality.

That changes what recursive improvement can accumulate. Instead of preserving only the latest agent implementation, a team can potentially preserve a history of empirically useful operating mechanisms and reuse them across subsequent search.

The paper does not establish that this process will continue improving indefinitely, nor that a plugin evolved on one workload will generalize broadly. It provides narrower evidence: within the tested settings, separating local mechanism discovery from global coordination produces stronger harnesses than the evaluated whole-system rewriting approaches, and some of those mechanisms remain useful beyond the configuration in which they were discovered.

That is enough to make the unit of optimization a serious architectural decision.

Cognaptus: Automate the Present, Incubate the Future.