When the Newest AI Skill Isn't the Best

TL;DR for operators

An AI agent can fix one visual-reasoning failure and introduce another. As its instructions and tools evolve, operators need a way to distinguish genuine capability improvements from revisions that merely solve recent examples. They also need to diagnose recurring failures without repeatedly examining entire execution histories.

VisualEvolve addresses both problems through versioned skill libraries and structured evidence retrieval. Across four frozen vision-language models and six evaluation sets, it improves over the initial skill library in 23 of 24 comparisons, with one tie. Removing historical version selection or structured evidence access substantially reduces performance. For organizations maintaining AI agents, the research supports treating skill revisions as evaluated, reversible artifacts rather than automatically accepting the latest changes. It does not yet establish enterprise reliability or lower total operating costs.

A better skill library requires more than another correction

Suppose a visual agent repeatedly misreads small text in a high-resolution document. Adding a cropping instruction may correct that error. But the same revision could encourage unnecessary cropping on other tasks, discard relevant context, or interfere with existing inspection strategies.

The relevant measure is whether the revised agent performs better across its responsibilities, not whether it fixes the example that prompted the change.

In VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning, Jiawei Wang and coauthors¹ investigate this problem by allowing agents to revise external skills while leaving the underlying vision-language model (VLM) unchanged.

The authors evaluate four frozen VLM backbones on six held-out evaluation sets drawn from V*Bench, HRBench, and VisualProbe, comprising 2,306 task instances.

Source: Section 4.2, Table 1. Macro-average Acc@1 is the unweighted average across six evaluation sets; pp denotes percentage points.

The largest individual improvement occurs on VisualProbe Hard with Qwen3.5-Plus, where accuracy rises from 39.6% to 54.7%, a 15.1-point gain.

All four backbones improve on their initial macro-average performance. Across the 24 individual backbone–evaluation-set combinations, 23 improve and one ties. None shows a reported decrease.

These results establish a benchmark advantage for the complete evolved libraries. They do not establish that every individual skill revision is beneficial.

Why VisualEvolve sometimes returns to an older version

VisualEvolve represents each skill as a textual strategy paired with an optional executable visual operation. Its initial library contains a cropping skill.

The framework repeatedly follows four stages: Select, Execute, Evolve, and Evaluate.

During each of 16 rounds, it selects a historical library, runs visual tasks, examines execution evidence, proposes skill modifications, and evaluates the updated library. Skills may be added, modified, or removed, including their executable operations.

The central design choice is how the next round begins.

Rather than automatically continuing from the newest library, Skill-Library Snapshot Selection chooses the archived version with the highest accuracy on a fixed 200-example validation set.

This matters because a revision that resolves one class of errors can degrade performance elsewhere. Preserving earlier configurations gives the evolving agent alternatives when a new revision performs poorly.

The component ablation provides direct supporting evidence. With Qwen3.5-Plus, removing Snapshot Selection reduces macro-average benchmark accuracy from 77.6% to 72.2%, a decline of 5.4 percentage points (Section 4.3, Table 2).

Removing the Execute stage reduces accuracy to 69.9%, while removing the Evaluate stage reduces it to 71.2%. These ablations assess the framework’s components under a common experimental setup, rather than comparing unrelated systems.

Snapshot Selection is therefore part of the improvement mechanism, not simply an administrative version-control feature.

It does not guarantee that a newly generated library improves. Moreover, repeatedly selecting against the same validation set creates a potential for selection-induced overfitting. The authors use separate held-out benchmarks for final evaluation, but repeated-run uncertainty remains incompletely characterized.

The second problem is knowing which failures deserve attention

Archiving agent revisions creates another difficulty: the evidence required to understand those revisions grows with every evolution round.

VisualEvolve addresses this through Progressive Evidence Access (PEA), which organizes accumulated information into four levels:

1. Skill-Performance Reports: indicate how correctness changes on validation examples associated with particular skills.
2. Evolution Logs: record historical skill modifications.
3. Trajectory Summaries: compress execution outcomes and diagnosed failures.
4. Raw trajectories: provide detailed execution evidence, retrieved selectively through references retained in the summaries.

The evolving agent can begin with broad performance signals and inspect progressively more detailed evidence when a particular failure requires investigation.

An important measurement distinction concerns the Skill Performance Score. It compares correctness between two library snapshots on validation examples where a particular skill was loaded in at least one snapshot.

A positive score indicates improvement on that associated subset. It does not establish that the skill itself caused the improvement: several skills may operate together, and other library changes can affect outcomes.

The ablations nevertheless show that structured evidence access contributes to the complete system.

With Qwen3.5-Plus, removing the Skill-Performance Report reduces macro-average accuracy by 4.0 points, removing Evolution Logs by 2.6 points, and removing Trajectory Summaries by 4.6 points. Removing PEA altogether reduces accuracy from 77.6% to 71.3%.

The evidence supports the combined diagnostic architecture under the tested conditions. It does not isolate the precise contribution of every retrieval decision.

For agent operators, the relevant design question becomes how to retain enough execution evidence for diagnosis while controlling what the evolving agent examines at each step.

The evolved capabilities go beyond rewriting instructions

The improvement mechanism is not confined to better prompts.

The appendix’s qualitative evolution analysis documents increasingly structured strategies for observation, visual search, text reading, counting, localization, and answer verification. Under the alternative CodeVision evolution setting, the framework also introduces executable rotation and flipping operations.

Reported accuracy on rotated validation examples rises from 64.3% to 73.2% after rotation is introduced. For flipped examples, accuracy rises from 42.3% to 80.8% following the addition of flipping.

These observations illustrate how diagnosed capability gaps can lead to new visual operations. They are evolutionary case evidence, not isolated causal estimates of rotation or flipping.

Two additional experiments examine the framework’s adaptability.

First, evolution using Default, Mini-o3, and CodeVision data produces macro-average gains of 8.8, 7.1, and 5.0 percentage points, respectively, over the initial library with Qwen3.5-Plus. This robustness test indicates that improvements are not confined to the default evolution corpus.

Second, a library evolved using Kimi-K2.5 transfers directly to three other backbones. Its macro-average accuracies trail backbone-specific evolution by 2.7 points on Qwen3.5-Plus, 1.4 on Qwen3.6-Plus, and 1.1 on Doubao-Seed-2.0-Pro.

This supports the possibility of reusable capabilities maintained outside model weights. Portability across all model combinations, however, is not demonstrated.

Fewer trajectories do not establish cheaper adaptation

VisualEvolve’s resource requirements are relevant because it changes external skills instead of updating model parameters.

One evolution run produces 3,912 Worker task-solving trajectories: 512 from round-specific execution tasks and 3,400 from initial and repeated validation.

The paper compares this figure with estimated reinforcement-learning rollout counts ranging from 245,760 to 819,200 for selected training-based visual-reasoning systems—approximately 63 to 209 times as many trajectories.

The counts are not equivalent measures of total computation. They do not equalize model architecture, trajectory length, training operations, data preparation, or additional model calls used to summarize evidence and evolve skills. Published training-based benchmark comparisons also use differing backbones and, in some cases, differing inference metrics.

The paper separately reports a mean API cost of $2.55–$4.12 per evolution round across evaluated backbones. On VisualProbe Hard, the evolved Qwen3.5-Plus library adds $0.0268 per evaluated example relative to its initial library while improving accuracy by 15.1 points.

For a production operator, that additional cost must be assessed against task volume, error consequences, latency, and the value of the accuracy improvement. Frozen weights do not mean unchanged inference expenditure.

What an enterprise agent team can reasonably adopt

Cognaptus infers a concrete management approach from these findings.

For teams operating agents on repetitive visual workflows, such as document inspection or image-based verification, the framework suggests maintaining versioned instructions and executable tools alongside their evaluation records. Recurrent failures could trigger targeted skill revisions, while independent regression tests determine which configurations remain eligible for use.

The approach is most relevant when tasks have verifiable outcomes, representative test examples are available, and the organization can preserve execution records with sufficient detail to diagnose errors.

The enterprise application remains an inference. VisualEvolve’s reported experiments concern visual-reasoning benchmarks, not production document-processing reliability, security, workflow latency, or downstream business losses.

A deployment test would therefore need to measure whole-workflow outcomes, including regression rates, inference overhead, tool-execution failures, and the cost of repeated validation. Historical snapshots provide a basis for rollback, but reliable production rollback and broader operational safeguards are not experimentally established by this paper.

Agent improvement becomes a selection problem

VisualEvolve’s contribution is not simply that AI agents can generate better skills without fine-tuning. Its experiments indicate that improvement depends materially on how revisions are selected and how execution evidence is organized for subsequent diagnosis.

The strongest supported result is better visual-reasoning benchmark performance from evolving external skill libraries under frozen backbones. The broader implication is an agent-maintenance architecture in which capability changes are versioned, evidence-linked, tested, and selectively retained.

For organizations considering self-improving agents, the research makes a credible case for investing in that architecture. Whether it delivers dependable improvement and favorable economics in a specific production workflow still requires direct measurement.

Cognaptus: Automate the Present, Incubate the Future.