Higher Pass Rate, More Broken Tasks: The Regression Tax in Agent Skill Libraries

TL;DR for operators

Adding reusable instructions to an agent creates a release-management problem that average accuracy does not fully expose. A library can solve tasks the baseline missed while simultaneously breaking tasks the baseline handled correctly.

Darshan Tank and Baran Nama measure that trade-off directly in The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents.¹ Across 18 skill-library conditions, they observe 553 baseline-fail-to-skill-pass transitions but also 324 baseline-pass-to-skill-fail transitions. The new failures offset 59% of the gross gains.

For an agent team, the practical implication is not “avoid skills.” Gross gains still exceeded regressions. It is to change the acceptance test. Report gains and regressions separately, test whether skill descriptions affect behavior before any skill body is invoked, and distinguish failures of input grounding, procedure, and output verification. In spreadsheet workflows, the verifier also deserves scrutiny: execution-based re-grading recovered 226 outputs whose formulas were already correct but had been scored as failures.

The boundary is substantial. These experiments cover office automation, each condition was run once per task, model and harness were coupled, libraries were not length-matched, and the proposed failure mechanisms come from observational trajectory coding rather than controlled causal interventions.

A better average can conceal lost capabilities

Suppose an agent passes 60% of a benchmark before a skill release and 65% afterward. A conventional evaluation records a five-point improvement. A deployment owner may reasonably conclude that the release made the system more reliable.

That conclusion is incomplete because the same five-point change can arise from very different task transitions.

For paired tasks, the paper writes the pass-rate change as:

where $G$ is the number of tasks newly solved, $R$ the number newly broken, and $N$ the paired task count.

The equation is elementary. The measurement choice is not. Looking only at $\Delta p$ discards the composition of the change.

Across the study’s 18 treatment cells, the libraries produced 553 gains and 324 regressions, leaving 229 net positive transitions. Every tested library condition broke at least some baseline successes, with regression counts ranging from 2 to 41.

This is what the paper calls the regression tax: not a new performance scalar, but the share of gross improvement consumed by newly introduced failures. Here, regressions cancel 59% of gains.

The statistical results further discourage reading every positive headline number as a reliable improvement. Only five of 18 net-effect comparisons reached nominal $p<.05$, and only three survived Bonferroni correction. All three corrected results came from Claude Code with Claude Sonnet 4.6 on SpreadsheetBench.

For release decisions, the unit of interest therefore shifts from “Did average accuracy rise?” to “Which capabilities were gained, which were lost, and are the losses acceptable?”

The failures often begin before the procedure

One tempting diagnosis is that a harmful skill simply contains a bad procedure. The OfficeQA-Pro trajectories point elsewhere.

The authors frame task execution as three stages: grounding, where the agent selects and interprets the relevant inputs; method, where it applies a procedure; and verification, where it checks the result.

Among 81 OfficeQA-Pro regressions, 59, or 72.8%, were coded as grounding displacement. Another 14, or 17.3%, were attributed to skill-description osmosis; three combined grounding and verification effects; five were classified as Other.

Grounding displacement means the added guidance changes a previously correct interpretation of the task or evidence. The procedure may be coherent once that interpretation is accepted. The failure occurred earlier: the skill redirected what the agent treated as the relevant input.

That matters because procedural libraries are primarily designed to improve the middle of the pipeline. If failures concentrate at the edges, adding yet more procedural detail may not address the limiting failure mode.

The mechanism evidence should be read as diagnosis, not causal proof. One author coded paired trajectories, there was no inter-coder reliability assessment, and the study did not experimentally isolate each content change. The paper appropriately presents these labels as candidate mechanisms.

A skill can influence the agent without being invoked

The study also identifies a subtler context channel. Skill descriptions may remain visible to the agent even when the full skill body is never opened or executed.

The authors call behavior changes attributable to this presence skill-description osmosis.

That distinction matters for systems that rely on retrieval or invocation gating as their main safety control. Disabling a skill body does not necessarily return the model to its no-library state if the description remains in context.

The paper codes 14 OfficeQA-Pro regressions as osmosis. In SpreadsheetBench’s coarser mechanism classification, 70 of 243 regressions fall into that category.

This does not establish that every invocation-free behavior change was caused by a description. The authors require additional influence evidence and assign unsupported cases elsewhere. Nor is the channel inherently harmful: a description could steer behavior beneficially as well.

Cognaptus inference: agent platforms should test at least three configurations when validating context-based skills: no library, descriptions only, and descriptions plus available bodies. That separates prompt-resident influence from the incremental effect of invocation more cleanly than a simple “skill enabled versus disabled” test.

Some spreadsheet failures belonged to the verifier

SpreadsheetBench exposes a second reliability problem: the measurement layer itself can misclassify agent output.

The original grader evaluates cell values, but some generated spreadsheet formulas require execution in a full spreadsheet engine before their resulting values can be judged. The authors therefore re-graded 663 formula-involved treatment failures using execution-based verification.

The result was not a wholesale reversal. 396 cases, or 60%, were genuinely wrong, and 41 remained unevaluable. But 226 cases, or 34%, contained already-correct formulas that the original grader had failed to recognize.

For Codex with GPT-5.4-mini, the effect on measured treatment performance was especially large. Depending on the skill library, raw SpreadsheetBench pass rates of 66.6–67.6% became roughly 77.8–79.1% after correction.

This re-grading serves as a verification test of the original measurement pipeline. It does not show that spreadsheet skill regressions disappear under better grading. It shows that part of apparent agent unreliability was produced downstream by a verifier unable to execute the artifact it was judging.

For spreadsheet agents—and more generally for agents that produce executable artifacts—the evaluator is part of the system being operationally validated.

Skill releases need a transition ledger

The study suggests a more informative release dashboard than one built around a single pass rate.

Cognaptus inference: approval criteria can then place explicit limits on regressions rather than letting them disappear inside net improvement. A ten-point gain obtained by replacing failures in low-risk workflows while breaking previously dependable financial or operational tasks is materially different from a ten-point gain with near-zero regressions.

The mechanism evidence also argues for three separate reliability investments. Document-oriented agents need stronger source selection, definition disambiguation, and input grounding. Procedural skills remain relevant to the method stage. Spreadsheet and other executable-output systems need environment-backed verification rather than shallow inspection alone.

This is broader than “write better skills.” It treats grounding, method, and verification as distinct engineering surfaces.

The evidence is strongest inside the tested workflow class

The paired outcome counts and spreadsheet re-grading results are directly observed within the experiment and are the strongest parts of the evidence.

Generalization requires more restraint. OfficeQA-Pro and SpreadsheetBench emphasize documents, tables, workbook state, formulas, and output validation. Domains in which procedural reasoning itself is the principal bottleneck could produce a different failure distribution.

The experiment also runs each condition once per task, so stochastic run-to-run variability is unknown. Each model is tied to its harness, preventing clean separation of model from harness effects. The treatment libraries differ in length and wording rather than constituting controlled token-dose variants. Finally, mechanism labels depend on observational trajectory comparisons rather than randomized content interventions.

Those constraints do not erase the central measurement result. They limit how confidently the observed mechanism proportions can be transported to other agent architectures and domains.

Reliability is about what changed, not only where the average landed

The paper’s most durable contribution is a modest change in bookkeeping.

When a skill library is introduced, compare every task with its own baseline outcome. Count the new successes. Count the newly broken successes too. Then investigate where those transitions entered the pipeline.

In this study, that view reveals three issues that an aggregate score compresses away: regressions consume a large fraction of gross gains, skill context can alter behavior without explicit invocation, and a weak verifier can make correct outputs look wrong.

For teams deciding whether an agent update is ready for production, the headline pass rate remains useful. It simply should not get the final vote.

Cognaptus: Automate the Present, Incubate the Future.