The Skill Library That Could Read but Couldn’t Run

TL;DR for operators

A system can discover agent “skills” that look coherent to humans and still fail to make an agent more capable.

That is the useful result of Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining.¹ The authors build a pipeline that cuts GUI interaction histories into segments, clusters those segments into candidate routines, generates explicit skill specifications, and trains a Qwen3-8B policy with Group Relative Policy Optimization, or GRPO, to compose the resulting skills.

The intermediate artifacts look promising. Five of eight discovered clusters achieve at least 0.95 purity against source-dataset skill labels. Several map neatly onto recognizable routines such as document editing, data transfer, file organization, and publishing.

Then the operational tests arrive.

On the synthetic source benchmark, GRPO raises skill-step accuracy from 18.5% to 20.5%. A trivial predictor that always selects the most frequent skill reaches 34.9%. A small Transformer reaches 34.6%. The GRPO policy records 0% exact sequence match.

Transfer is no kinder. Relative to zero-shot Qwen3-8B, the trained policy falls from 55.8% to 44.2% on WebArena and moves from 43.5% to 43.3% on BrowseComp+. There is no completed GRPO evaluation on Mind2Web and no live WorkArena result.

The business conclusion is therefore specific:

• Useful now: mining workflow traces for documentation, taxonomy discovery, transition analysis, audit support, and human-reviewed procedure drafting.
• Not demonstrated: autonomous extraction of a portable skill vocabulary that improves computer-use policies across environments.
• Minimum validation standard: frequency priors, transition-memory priors, and a modality-matched supervised model must be tested before an elaborate learned system receives credit for anything.
• Main technical lesson: an interpretable cluster is a descriptive artifact. A reusable skill is a causal unit that helps an agent choose and execute the next action under changing conditions. Those are not the same product.

A method that loses to “predict the most common thing” has not yet earned a deployment roadmap. It has, however, earned our attention for showing exactly where the roadmap breaks.

The expensive policy loses to “most common”

The strongest result in this paper is not the clustering score. It is the baseline failure.

The authors evaluate whether their mined skill vocabulary helps models predict sequences of skills on held-out workflows. The central source-domain comparison is stark:

Lower edit distance is better. The GRPO policy is not merely behind the frequency baseline on accuracy. Its predicted sequences require considerably more correction.

That matters because the frequency predictor contains no sophisticated representation of workflow structure. It simply exploits class imbalance: when some skills occur much more often than others, predicting the most common skill becomes surprisingly competitive.

“Surprisingly” is doing charitable work here. Class imbalance is not an obscure adversarial condition. Repetitive enterprise processes are full of it. Users search more often than they publish, click more often than they resolve exceptions, and perform standard updates more often than rare recovery procedures. Any workflow-learning system that cannot improve upon those regularities has not yet shown that it learned the workflow.

The small Transformer nearly matches the frequency baseline, suggesting that sequence context contains some usable signal. But the large language model trained through GRPO does not capture it effectively under the paper’s setup. Model size, apparently, is not a substitute for aligning the representation, objective, and evaluation. The industry will recover from this revelation.

The paper tests a promotion, not merely a pipeline

The paper’s three stages can look like one continuous technical method:

1. Detect skill boundaries in action trajectories.
2. Cluster the resulting segments into named routines.
3. Train a policy to select and compose those routines.

But each stage makes a different claim.

The first stage asks whether visible changes between adjacent GUI actions indicate where one skill ends and another begins.

The second asks whether the resulting segments contain enough regularity to form an interpretable taxonomy.

The third asks whether that taxonomy improves downstream decisions.

Only the third stage establishes operational value. A successful boundary detector does not prove that the segments are useful. A clean clustering result does not prove that its categories are executable. A readable Markdown file does not prove that a policy can apply it.

The study is therefore best understood as a promotion process:

Repeated action pattern
        ↓
Candidate segment
        ↓
Coherent cluster
        ↓
Human-readable skill
        ↓
Predictively useful skill
        ↓
Transferable policy primitive

The pipeline advances comfortably through the middle of this chain. It does not reach the bottom.

This is why the paper is more valuable as a diagnostic negative result than as another agent-framework announcement. It separates the ability to describe behavior from the ability to control behavior.

Boundary detection finds changes, not necessarily skills

The first phase represents each primitive GUI action with 15 features, including action type, screen coordinates, timestamp, text length, and scroll amount. It then places a boundary where the Euclidean distance between adjacent action vectors exceeds a threshold selected on the source dataset.

On InteraSkill Workflows, or IW, the selected threshold produces:

• precision of 0.419;
• recall of 0.803;
• F1 of 0.538.

The high recall is encouraging. Most real skill transitions are accompanied by noticeable changes in action features.

The low precision reveals the problem. Noticeable changes also occur inside skills.

A user may click a field and then type into it. They may scroll while reviewing a document. They may copy a value, switch applications, and paste it elsewhere. These action changes are not noise; they are often the internal grammar of the routine.

The detector therefore confuses action discontinuity with skill discontinuity. It finds many legitimate boundaries, but it also cuts coherent procedures into fragments.

That distinction is easy to miss in workflow analytics. A large change in observable behavior is not automatically a change in intent. Consider a customer-support representative who searches for a ticket, opens it, changes several fields, adds a note, and saves. The coordinates, text lengths, and action types vary throughout. Operationally, the sequence may still be one routine: update the ticket.

The sensitivity test in the appendix shows that the IW F1 score remains fairly stable when the threshold moves between the 40th and 60th percentiles. This is a robustness check on source-domain threshold selection. It shows that the reported IW score is not the product of a single exquisitely convenient cutoff.

It does not establish cross-domain robustness.

When the IW-derived threshold is applied to WebArena, precision becomes 1.000, recall falls to 0.100, and F1 falls to 0.119. A threshold tuned using WebArena’s own labels can reach F1 of 0.851, but that is an oracle diagnostic. It proves that an appropriate threshold exists after observing the target labels. It does not prove that the source calibration transfers.

For an enterprise deployment, this means boundary rules cannot be treated as universal preprocessing. Different applications have different interaction scales, interface conventions, action distributions, and workflow rhythms. A threshold learned from synthetic office routines may not segment map navigation, a CRM, a spreadsheet, and a service portal in comparable ways.

Before mining reusable skills, the system must first answer a more basic question: reusable across what action geometry?

The clusters are readable because some workflows have distinctive fingerprints

After segmentation, each candidate segment is compressed into the mean and diagonal variance of its action features. The result is a fixed-size representation of the segment’s action inventory.

This representation is cheap and length-invariant. It can recognize that two segments contain similar proportions of clicking, typing, scrolling, copying, or saving even when the segments have different lengths.

It also discards action order.

A sequence containing copy → switch application → paste may receive a similar statistical summary to a reordered sequence containing paste → switch application → copy. One is a plausible data-transfer routine. The other is a small procedural disaster.

The authors cluster these segment summaries using a diagonal Gaussian Wasserstein distance. On IW, eight clusters produce normalized mutual information, or NMI, of 0.650 and purity of 0.628. Increasing the number of clusters does not improve NMI, so eight is selected for the main analysis.

A supervised-contrastive encoder is then trained using the cluster assignments as pseudo-labels. In the refined 16-dimensional space, KMeans reaches:

• NMI of 0.862;
• silhouette score of 0.554;
• purity of 0.837.

The NMI improvement is substantial: roughly 33% relative to the original Wasserstein clustering result. The contrastive training curve is stable, and the appendix visualization shows separated source-domain groups.

This is genuine evidence that the pseudo-labels contain internally learnable structure.

It is not evidence that the structure is a transferable action language.

Five of the eight clusters have purity of 1.00 against a dominant IW skill:

These are readable because their action inventories are distinctive.

Other categories are much less clean. The cluster dominated by send_message has purity of 0.28. The review_content cluster reaches 0.46. Both are built from generic combinations of clicking, typing, and scrolling that occur in many tasks.

The representation can therefore separate workflows whose primitive-action fingerprints differ strongly. It struggles when multiple intents share the same ingredients.

This is not unusual. A bag of words can distinguish a legal contract from a restaurant review because their vocabularies differ. It is less useful for distinguishing “approve the payment” from “reject the payment” when both messages mention the same invoice, vendor, amount, and account.

The missing information is relational: order, state, object identity, preconditions, and intended outcome.

Readability is not an independent validation result

The paper includes qualitative natural-language descriptions of the discovered clusters. These descriptions are useful, but their evidentiary role needs careful handling.

Claude Haiku 4.5 generated the descriptions after receiving:

• the cluster’s top action types;
• its dominant ground-truth skill label;
• its purity;
• exemplar segments near the cluster centroid.

The model therefore saw the plurality human label before describing the cluster.

This is not a blind semantic test in which an independent evaluator inferred the skill meaning from raw behavior alone. It is a concise verbalization of already computed cluster statistics.

That does not make the descriptions invalid. It makes them documentation, rather than additional proof that the taxonomy has independently recovered human concepts.

This distinction matters for businesses evaluating automatically generated process maps. A polished label such as “record update,” “customer communication,” or “content review” can make a cluster feel more semantically complete than it is. Natural-language naming smooths over mixed membership, missing order information, and ambiguous intent.

Readable labels are valuable for inspection. They should not be allowed to launder an action motif into an executable business capability.

The reward model learns resemblance to IW, not success in the world

The policy stage uses Qwen3-8B and GRPO. The policy receives task context and mined skill names, then generates a skill-aware response containing a proposed skill sequence and next-action field.

The reward model is another Qwen3-8B model with a scalar classification head. It is trained on pairwise preferences derived from 1,275 IW training conversations and 225 validation conversations.

Ground-truth skill flows serve as preferred answers. Rejected plans are created synthetically through operations such as:

• swapping adjacent skills;
• replacing a skill;
• deleting or inserting a skill;
• globally reordering the plan;
• duplicating or mutating elements.

The preference margin is derived from a heuristic score combining:

This objective teaches the reward model to recognize plans that resemble IW’s annotated skill flows.

It does not teach the reward model whether a browser task succeeds.

The reward is not trained using WebArena completion, BrowseComp+ correctness, live WorkArena state, Mind2Web action success, or any target-domain task outcome. It is a source-domain plan-similarity model.

That creates a clean explanation for the transfer failure. GRPO can optimize the proxy it receives without learning what deployment requires. The policy is rewarded for producing IW-like sequences, while the target environments may organize behavior differently, use different interfaces, expose different state, or require different boundaries between routines.

This is not evidence that GRPO is broadly unsuitable for computer-use agents. The paper does not make that claim, and neither should anyone else. Online reinforcement-learning systems that receive environment feedback are solving a different problem.

The result is narrower and more useful: offline optimization against a source-specific structural proxy does not automatically yield cross-domain task competence.

An organization building workflow agents should translate this into a reward-design question:

Does the reward measure whether the proposed routine resembles prior documentation, or whether it actually produces the required business state?

The former can help with formatting and process conformance. The latter is needed for reliable automation. They may correlate within a stable workflow. They should not be assumed equivalent across systems.

Transfer removes the comforting interpretation

The source-domain result might still be defended as early evidence: perhaps the model needs more training, or perhaps 20.5% accuracy is a modest first step.

The held-out checks make that interpretation harder.

The WebArena decline is substantial. BrowseComp+ is effectively unchanged. WorkArena-NLP shows no improvement and 0% exact match for both Qwen variants.

WorkArena-NLP is deliberately text-only. It converts enterprise task schemas into natural-language goals and asks the model to produce structured JSON plans. It removes screenshots, browser state, menu interaction, clicking, form filling, session handling, and interface recovery.

Its purpose is diagnostic: can the model reconstruct the intended structured workflow from text alone?

It is not a proxy for live computer use. A system can understand that a record should be filtered and updated without being able to locate the correct UI control, preserve state, submit the change, or verify the outcome.

Similarly, Mind2Web appears only as a zero-shot diagnostic. Base Qwen3-8B reaches 9.5% task completion, 28.8% action accuracy, and 33.8% element accuracy. There is no completed GRPO result against which to compare those figures.

These omissions define the paper’s claim boundary:

• verified transfer evidence exists for WebArena and BrowseComp+ skill-sequence checks;
• no positive improvement appears there;
• live enterprise transfer has not been tested;
• primitive action-level transfer after GRPO has not been established.

That is enough to reject the broad claim that the mined vocabulary currently functions as a portable skill library. It is not enough to conclude that no improved version could.

Auto-generated manuals beat a weak manual, then lose to frequency

The study also compares auto-generated SKILL.md specifications with a hand-crafted transition table and simple statistical baselines.

The generated artifacts contain descriptions, transition probabilities, workflow patterns, validation suggestions, and error-handling guidance. Qualitative examples cover service-ticket updates, spreadsheet-to-CRM transfer, and shared-drive file organization.

These examples reveal a plausible near-term use. The generated specifications can turn repeated traces into draft operational documentation. They bind recurring action motifs to task-specific fields and produce an inspectable starting point for an engineer or process owner.

The quantitative test asks whether the generated specifications improve next-skill sequence prediction as the amount of IW training data changes.

The result is mixed against the hand-crafted table. Auto-SKILL.md performs better at some dataset sizes and worse at others.

Against the frequency baseline, the result is consistent: frequency has lower normalized edit distance at every evaluated size.

At the largest reported setting:

• Frequency edit distance: 0.485
• Auto-SKILL.md edit distance: 0.528
• Hand-crafted table edit distance: 0.569

The generated specification beats the simple manual table there, but it still loses to the most-common-skill predictor.

This comparison should not be read as proof that automated documentation is useless. The hand-crafted baseline is a simple transition table, not a deeply engineered expert system. The generated files may still reduce documentation effort or reveal missing patterns.

It does show that a more detailed-looking artifact does not necessarily encode better predictive structure. Documentation richness and decision usefulness remain separate dimensions.

The appendix supports one thesis and blocks several others

The appendix contains many additional runs, sweeps, visualizations, and examples. Their likely purposes differ.

The distinction around the MLP, Transformer, and GRPO comparison is especially important.

The models do not receive the same inputs or use the same training objective. The MLP and Transformer consume continuous embeddings and are trained directly for next-skill prediction. Qwen3-8B consumes text prompts and is optimized through a learned reward model.

The table is therefore not a clean ablation proving that one architecture or the reward model causes the performance gap. It is a pipeline-level sanity check. Every variant is supposed to benefit from the mined skills; several do not beat simple priors.

A causal diagnosis would require controlled comparisons such as:

• Qwen3-8B with supervised next-skill training on the same prompts;
• the embedding policy with and without the same reward;
• ground-truth boundaries substituted while later stages remain fixed;
• order-aware and orderless segment encoders under the same training setup;
• source-only versus mixed-domain reward training;
• task-success rewards versus skill-flow similarity rewards.

Until those controls exist, the paper identifies plausible failure mechanisms rather than isolating a single culprit.

The business value is cheaper inspection, not autonomous transfer

Cognaptus draws three practical inferences from the evidence.

Trajectory mining can reduce process-discovery effort

Organizations already possess large quantities of interaction data: audit logs, browser traces, robotic-process-automation recordings, help-desk sessions, screen events, and workflow histories.

Mining repeated subsequences from those records could help identify:

• common operational motifs;
• dominant transitions;
• high-frequency workflow paths;
• ambiguous or overloaded procedure categories;
• processes that deserve formal documentation;
• exception patterns requiring manual review.

The paper demonstrates that action traces can produce readable source-domain clusters, particularly when workflows have distinctive action signatures. That is useful for discovery and documentation even when the resulting taxonomy is not ready to control an autonomous agent.

Generated skill files can become reviewable drafts

The auto-generated SKILL.md examples include reusable steps, task bindings, validation suggestions, and cautions. These are not equivalent to expert procedures, but they can provide scaffolding.

A sensible business workflow would be:

Interaction traces
        ↓
Automatically proposed routines
        ↓
Process-owner review
        ↓
Validation and recovery rules added
        ↓
Approved skill specification
        ↓
Controlled evaluation
        ↓
Limited deployment

This approach treats automatic generation as documentation assistance rather than policy authority.

It is less glamorous than announcing self-improving agents. It is also much closer to something a compliance team might permit.

Transition priors are useful whether or not they are intelligent

The paper’s baselines expose strong regularity in workflow sequences. Frequency and bigram-transition models can be highly competitive because business processes are repetitive by design.

That regularity is operationally valuable. A transition table can:

• detect unusual next steps;
• flag deviations from standard procedures;
• prioritize likely actions in a human interface;
• recommend candidate routines;
• provide fallback behavior when a learned policy is uncertain;
• establish a transparent benchmark for more complex models.

The baseline should not be dismissed because it is simple. Simplicity is precisely why it is useful: its behavior is inspectable, cheap, and difficult to flatter with a conveniently chosen metric.

Every mined-skill proposal needs three controls

The paper’s most transferable recommendation is methodological. Any system claiming that mined skills improve an agent should be tested against at least three controls.

1. Frequency prior

Always predict the most common skill, globally or within a relevant context.

This tests whether apparent learning is merely exploitation of class imbalance.

2. Transition-memory prior

Predict the next skill from a simple transition table, bigram model, or workflow-state count.

This tests whether the task can be solved through shallow sequential regularity without semantic skill representations.

In the paper’s appendix, an AWM-style transition baseline reaches 78.8% mean position accuracy on WebArena, far above several large-model results. WebArena’s homogeneous navigation traces make transition memory unusually effective. That is precisely why the control matters.

3. Modality-matched supervised policy

Train a model on the same inputs and outputs as the proposed policy using direct supervised loss.

This tests whether reinforcement learning, a reward model, or an elaborate optimization loop contributes value beyond ordinary prediction training.

The paper lacks this fully matched control for Qwen3-8B. Without it, one cannot isolate whether GRPO, the reward model, the text representation, or their combination is responsible for the weak outcome.

A procurement or research review should not accept “our learned system beats zero-shot” as sufficient evidence. The relevant question is whether it beats the strongest cheap explanation of the data.

The next system must preserve order, state, and outcome

The paper suggests several concrete improvements, and the results indicate why each matters.

Replace action-jump boundaries with learned segmentation

A learned detector could use action-prediction error, object changes, semantic state transitions, or task context to distinguish internal action changes from genuine routine boundaries.

The objective should not be to find every visual discontinuity. It should identify points where the agent’s subgoal changes.

Encode sequences rather than action inventories

Means and variances are useful for clustering, but GUI routines are procedural. An order-aware encoder should represent:

• which action precedes which;
• which UI object each action targets;
• whether a value has been copied or entered;
• the state before and after the segment;
• preconditions and termination conditions;
• whether the intended result was verified.

A skill is not merely a collection of verbs. It is a state transition with rules.

Train rewards on outcomes

If the downstream objective is successful workflow execution, the reward should incorporate task completion, environment state, validation checks, or human judgments tied to business correctness.

Similarity to a reference plan may remain one component. It should not be mistaken for the outcome itself.

Mix source and target distributions

A vocabulary mined entirely from one synthetic domain will naturally encode its action frequencies, labels, segmentation scale, and procedural assumptions.

Cross-domain training should expose the representation and policy to varied interfaces and task structures. Otherwise, “general skill” becomes a flattering name for “source-dataset regularity.”

Run factorial evaluations

Each pipeline stage should be replaced independently while the others remain fixed. This is necessary to determine whether failures originate in segmentation, representation, clustering, prompting, reward modeling, optimization, or evaluation.

Without controlled substitutions, the diagnosis remains architectural impressionism: evocative, plausible, and difficult to invoice.

The current evidence has four hard boundaries

The practical interpretation should remain within four limits.

First, IW is synthetic. Its 2,000 fabricated trajectories cover enterprise-style routines, but they do not reproduce the full ambiguity, interface drift, exception handling, permission constraints, latency, interruptions, and undocumented behavior of real organizations.

Second, the main policy metrics measure skill-sequence prediction, not successful end-to-end GUI execution. Correctly naming a routine is not the same as completing it.

Third, the reported benchmark percentages are single-run results. They establish the observed outcome for the tested configurations, not a stable performance distribution across seeds, prompts, data samples, or training schedules.

Fourth, the most relevant enterprise transfer evaluation is absent. WorkArena-NLP removes the browser. Live WorkArena has not been run, and the trained policy has not been evaluated on Mind2Web.

These boundaries do not weaken the paper’s negative conclusion. They limit its scope.

The study directly shows that this particular pipeline does not convert readable source-domain clusters into a policy that beats simple priors or transfers positively on the completed checks.

It does not show that trajectory-mined skill libraries are impossible, that GRPO cannot train GUI agents, or that explicit skills have no value. Those larger claims remain open.

A skill library must earn the word “skill”

The appealing story behind automatic skill discovery is straightforward. Observe enough successful work, discover the repeated routines, write them into reusable files, and train an agent to compose them. Manual engineering recedes; organizational behavior becomes data.

The paper tests that story rather than merely illustrating it.

Its pipeline finds recognizable patterns. It labels them. It turns them into readable artifacts. It trains a large policy around them.

Then a frequency table wins.

That loss is not an embarrassment to be hidden behind clustering visualizations. It is the result that converts the paper from a demonstration into a useful evaluation study.

For operators, the near-term opportunity is not autonomous skill induction. It is assisted process discovery: use interaction traces to propose routines, expose transition structure, identify candidate documentation, and support human review.

For researchers and platform builders, the requirement is stricter. A reusable agent skill must preserve order, connect actions to state changes, survive domain shifts, and improve decisions beyond the regularities already available to a counting table.

Readable is a property of documentation.

Reusable is a property of performance.

The industry would benefit from checking which one it has built before naming the directory.

Cognaptus: Automate the Present, Incubate the Future.