Approved One by One, Risky in Combination: Testing Agent Skills Before They Execute

TL;DR for operators

A marketplace can approve several useful skills independently, then have an agent activate them together for an ordinary business task. The problem is that individually acceptable components can combine their preconditions, resource changes, and outputs into a plan-level objective that neither the user nor any single skill requested.

The reported benchmark shows why this interaction needs separate testing: severe plans rise from 4.7% of single-skill compositions to 66.5% of five-skill compositions. This does not prove that adding skills directly causes harm in every setting, but it does show that standalone approval is not enough evidence for composition safety.

SkillFuzz provides a pre-execution screening layer. It compares the plan generated with a skill composition against the plan for the same task without those skills, then prioritizes combinations that produce severe and previously unseen changes.

For operators, the implication is practical: keep reviewing individual skills, but allocate admission testing, regression screening, sandbox capacity, and human review to compositions as well. Use planning-layer screening to rank cases, execute the strongest flags in controlled environments, and rerun the tests whenever skills, planners, policies, or representative tasks change.

Independent approval does not establish composition safety

A marketplace approves several extensions independently. One can analyze a document, another can transform media, a third can create files, and a fourth can call external tools. Each appears compatible with its stated purpose. An agent then activates several of them for an ordinary business task.

The combined instruction context can introduce an objective that the user did not request and that no individual skill explicitly prescribed. One skill’s postcondition may satisfy another skill’s precondition. A resource created by one component may become an input to another. Separately acceptable actions can therefore form an unacceptable workflow.

The paper calls such a composition-induced objective an implicit intent. SkillFuzz¹ treats it as a property of the task-conditioned composition rather than evidence that one component is inherently malicious.

The composition-depth result makes the concern measurable. Using the paper’s fixed severity threshold, the share of severe plans rises from 4.7% for singleton skills to 66.5% for five-skill compositions. This analysis supports the paper’s claim that risk can grow through interaction rather than simple addition.

It does not estimate the causal effect of activating one more skill. Larger compositions may be sampled differently by the search procedure, and the result comes from one benchmark environment. Even with that boundary, an operator cannot reasonably use per-skill approval as the sole control once agents can combine skills dynamically.

The affected decision is concrete: a marketplace safety team deciding which releases, updates, or combinations require further testing. The relevant condition is multi-skill activation within a shared planning context. The boundary is that composition testing supplements rather than replaces component review.

SkillFuzz uses plans as a pre-execution warning surface

Realistic execution testing is expensive. It may require credentials, live APIs, proprietary data, external services, destructive permissions, or a faithful replica of the production environment. Exhaustively executing combinations from a marketplace of 196 skills would also be combinatorially impractical.

SkillFuzz moves the first screening stage to the planning layer.

For each task, it first obtains a baseline plan without activated skills. It then generates another plan with a selected skill composition. The semantic distance between the two plans becomes a measure of plan drift:

Here, $\pi_0$ is the no-skill plan and $\pi_{\mathbf{s}}$ is the plan produced under composition $\mathbf{s}$. High drift means the declared workflow has moved materially away from what the same planner proposed before the skills were introduced.

This is a differential signal, not a calibrated probability of harm. A large change may be legitimate, and a small textual change may conceal a consequential action. Its value comes from prioritization: it gives operators a scalable surface for deciding which combinations merit deeper inspection.

The framework also converts each natural-language skill document into a structured contract containing preconditions, postconditions, resource modifications, invariants, domain scope, and abstract actions. These contracts expose interaction pathways that raw skill names or generic embeddings may miss.

After pruning the marketplace to task-relevant candidates, SkillFuzz uses Monte Carlo tree search to explore compositions. Its contract-guided mutation step looks for an inactive skill whose contract is semantically close to the current risk-directed target, such as an unresolved postcondition or a potentially violated invariant.

The reward combines drift with novelty:

Intent Coverage Quality contributes reward when a composition produces a semantically new intent, with the reward scaled by the magnitude of the deviation. ICQ should therefore be read as a search objective: it favors findings that are both different from previous discoveries and operationally substantial.

Better testing is not the same as testing more pairs

The most decision-relevant comparison is the paper’s fixed-budget ablation. Six strategies receive 1,000 composition queries on the same SEC financial-report task with the same DS-R1-7B planner.

Random search finds five more distinct intents and exercises far more possible skill pairs. If the objective were broad combinatorial sampling, random search would look stronger.

SkillFuzz instead finds 90 high-severity intents, compared with 64 for random search—a 41% increase under the same query budget. Its mean drift is also higher, and 77% of its discovered intents cross the severity threshold, compared with 52% for random search.

The practical lesson is not that coverage is irrelevant. It is that coverage metrics encode different operational goals. A marketplace performing an exploratory audit may value broad pair coverage. A team with limited sandbox capacity may instead prefer a search policy that sends a higher proportion of severe, novel cases into the escalation queue.

Removing contract guidance reduces high-severity discoveries from 90 to 55 and cumulative ICQ from 66.7 to 49.1. This ablation is especially informative because it tests the purpose of the structured representation: the contracts do not merely make findings easier to describe afterward. They materially redirect where the search budget is spent.

The comparison remains narrow. It is reported for one task and one planner, so it does not demonstrate that the same ranking will hold across every domain or model. Marketplace operators should reproduce the ablation on their own task distribution before choosing a search policy.

Selected plan-level flags survive independent execution

Planning artifacts are useful only if they provide some information about behavior. The paper tests this connection by selecting 98 high-risk flagged compositions and executing them independently in sandboxed Docker environments.

A fixed Claude-based executor receives the raw skill composition, replans without seeing the original predicted intent, and performs the workflow. A separate judge then checks whether the predicted side effect appears in the execution trace.

Observable effects are detected in 80.6% of the selected cases. The rates are nearly identical across planner families: 80.5% for DS-R1 planners and 81.0% for GPT planners.

This is the paper’s main evidence that the testing layer finds more than unusual wording. Many high-risk plan deviations correspond to trace-level side effects under a separate executor.

The scope must remain intact. These were 98 selected high-risk flags, not a representative sample of all 1,188 discovered intent texts. The study uses one executor and an LLM-based trace judge. It therefore does not estimate marketplace-wide sensitivity, specificity, or incident probability.

The unconfirmed cases are also informative. They are concentrated in diffuse changes to analytical framing, which are harder to map to discrete events than file creation, media modification, or API invocation. Confidence in a plan-level flag should consequently depend on the type of predicted effect. Concrete resource and tool actions are easier to validate than subtle shifts in analysis or emphasis.

Marketplace testing should become a staged escalation workflow

The paper directly demonstrates benchmark performance. The following deployment design is a Cognaptus inference from that evidence.

This workflow changes the unit of governance. Admission review remains skill-specific, but release confidence becomes conditional on representative compositions and tasks. A skill update can invalidate earlier results even when its standalone purpose has not changed, because its new postconditions or resource behavior may interact differently with existing components.

The paper’s four descriptive clusters—unsanctioned data analysis, audio or video side effects, covert resource creation, and unauthorized tool invocation—can also help structure escalation queues. Operators should treat them as benchmark-derived labels rather than a complete ontology. A live marketplace may require additional categories for financial actions, access-control changes, disclosure failures, or domain-specific compliance rules.

The method is a triage layer, not a safety certificate

Several uncertainties affect deployment.

First, the plan oracle observes declared intent. An agent may omit, obscure, or depart from its stated plan. Selective execution validation reduces this concern for the tested high-risk cases but cannot eliminate false negatives.

Second, drift, novelty, relevance, and severity thresholds are globally fixed in the benchmark. A healthcare workflow, a coding assistant, and an internal document agent do not share the same acceptable deviation. Thresholds need calibration against local tasks, permissions, and loss scenarios.

Third, the evidence comes from the 196-skill SkillsBench marketplace, ten selected tasks, eight planners, and one fixed execution protocol. The cross-agent results show that nonzero composition risk is not confined to one tested model family, but they do not establish prevalence in a production marketplace.

Finally, low plan drift should not be converted into approval. A composition may preserve the wording of a plan while changing tool arguments, execution order, hidden state, or permission use. SkillFuzz is strongest as a mechanism for finding cases worth escalating, not for certifying the remainder as safe.

Composition deserves its own test budget

The paper changes a practical assumption behind skill-marketplace governance. Safety cannot be fully inherited from individually approved components when an agent can assemble those components into a new instruction context.

SkillFuzz offers a workable response: use structured contracts and differential planning to search composition space before expensive execution, then direct sandbox and human-review capacity toward severe and novel deviations. Its strongest benchmark result is not exhaustive coverage. It is a more risk-directed allocation of a fixed query budget.

For marketplace operators, the next control is therefore not another standalone skill checklist. It is a repeatable composition-screening pipeline tied to release, update, planner, and task changes—with execution reserved for the cases where planning evidence gives the clearest reason to look closer.

Cognaptus: Automate the Present, Incubate the Future.