From YouTube to Execution: How GUIDE Teaches AI Agents to Actually Use Software

Tutorials are where software knowledge goes to become useful, messy, and mildly unbearable.

A human trying to learn GIMP, LibreOffice Calc, Thunderbird, or VS Code can survive this mess. We search YouTube, skim a video, ignore the creator’s life story, watch the cursor, and remember that the menu item we need is not where our intuition said it would be. A GUI agent, even a strong vision-language model, has a harder time. It may see the screen. It may understand the instruction. It may even know the general category of action. Then it clicks the wrong menu because the software has its own local customs. Software, regrettably, has culture.

That is the useful starting point for GUIDE, short for GUI Unbiasing via Instructional-Video Driven Expertise. The paper proposes a training-free framework that retrieves web tutorial videos at task time, extracts procedural knowledge from them, and injects that knowledge into existing GUI agents without changing model weights or architecture.¹

The important word is not “video.” The important word is “expertise.” GUIDE is not merely giving an agent more pixels to stare at. It is converting tutorial videos into two kinds of domain memory: planning knowledge about what steps are needed, and grounding knowledge about where the relevant interface elements are likely to appear.

That distinction matters because the common misconception is too simple: if GUI agents fail, perhaps they need better visual perception. Sometimes yes. But the stronger diagnosis in this paper is that many failures are not eye problems. They are workflow problems. The agent knows what brightness means, but not that GIMP puts contrast adjustment under Colors, not under the kind of image menu it may expect from other software. The model is not stupid. It is just insufficiently socialized into GIMP society. A tragic fate, but apparently fixable.

The bottleneck has two names: planning bias and grounding bias

GUIDE frames GUI-agent domain bias as two related but separable problems.

Planning-level bias is about workflow. The agent does not know the correct sequence of operations inside a specific application: which menu to open, which dialog to expect, which stage comes next, and which tempting path is actually a dead end. This is the “what should I do next?” failure.

Grounding-level bias is about interface localization. The agent may know that it needs a contrast slider, a formatting menu, or a file export option, but it cannot reliably identify the relevant visual element in the current screenshot. This is the “where exactly is the thing?” failure.

The paper’s mechanism-first contribution is to treat these two failures differently. GUIDE does not dump a tutorial transcript into the prompt and hope that the model becomes enlightened. It retrieves a task-relevant video, processes the video into structured annotations, splits the resulting knowledge into planning and grounding channels, and routes each channel to the part of the agent architecture where it can actually help.

A compact view of the mechanism looks like this:

This is the article’s core argument: GUIDE is interesting not because it watches videos, but because it builds a small operational supply chain from public tutorial material to executable agent guidance.

Subtitles are the retrieval layer, not a decorative transcript

The retrieval stage begins with a practical annoyance: tutorial video titles are noisy. A title like “Excel Tips 2026” or “GIMP Tutorial for Beginners” may or may not contain the operation the agent needs. Titles are marketing. Subtitles, by contrast, often contain actual procedural language: “click the Colors menu,” “select the cell range,” “open the format dialog,” and so on. This is not elegant, but it is useful. In automation, useful usually wins.

GUIDE uses subtitles as a semantic bridge between the task instruction and the content inside the video. The pipeline begins with YouTube candidate retrieval, then applies metadata filtering, subtitle-assisted GUI-domain classification, subtitle-driven topic extraction, and topic-title relevance matching.

The three-stage narrowing is important:

1. Domain classification filters out videos that are not real GUI demonstrations, such as lectures, reviews, entertainment, or non-operational content.
2. Topic extraction distills what the video actually demonstrates, using subtitles and title together rather than trusting the title alone.
3. Relevance matching scores whether the extracted topic matches the current task, retaining the top video and allowing additional videos only when relevance is high enough.

This retrieval design tells us something useful for enterprise automation. The input corpus does not have to be clean if the filtering layer is strict enough. Internal SOP videos, training recordings, onboarding demos, and support walkthroughs are rarely organized like a dataset. They are closer to YouTube than to a benchmark: inconsistent, verbose, duplicated, and full of side remarks. GUIDE’s bet is that narration can still expose operational semantics.

The paper’s human evaluation supports the direction of that bet. On 300 sampled candidate videos, the subtitle-assisted GUI classifier achieves 100% precision and about 91% recall. In plain English: the retained videos are clean, while some valid videos are missed. That is a conservative trade-off. For a GUI agent, a missed tutorial usually means falling back to baseline behavior. A wrong tutorial can actively derail execution. The system prefers silence over bad advice, which is a refreshing policy. Some humans could try it.

The topic extraction stage also looks usable rather than ornamental: on 300 confirmed GUI videos, the mean human score is 0.867, and 96% of extracted topics are rated acceptable. The boundary is equally clear. Subtitle quality matters. Visual-only demos or poor auto-captions are weaker inputs. GUIDE is not magically understanding all video. It is exploiting a particular affordance of tutorial media: people narrate what they are doing.

Inverse dynamics turns screen changes into transferable procedure

Once GUIDE has a relevant tutorial video, it still faces a harder problem. Video is not directly actionable. A GUI agent does not need a movie; it needs something closer to procedural memory.

GUIDE’s annotation pipeline uses an inverse dynamics idea. Instead of predicting the next screen from an action, it compares consecutive interface states and infers what action likely happened between them. The pipeline transcribes audio with Whisper, extracts keyframes using visual-change detection aligned with subtitle intervals, parses UI elements using OmniParser, and sends keyframe pairs plus element graphs, video topic, and local subtitle context to a VLM annotator.

The output is not merely “click at coordinate $(x, y)$.” That would be brittle. Coordinates change across screen sizes, versions, themes, and window positions. GUIDE instead produces natural-language descriptions of meaningful actions, including the likely intent, the relevant visual elements, and the reasoning behind the step.

This is where the method becomes more interesting than ordinary demonstration replay. A coordinate trace says, “click here.” A transferable annotation says, “open the Colors menu because this application places brightness and contrast controls there.” The second form can survive layout variation. The first is a souvenir from someone else’s monitor.

The paper also adds a “Meaningful” filter to remove transitions that are visually different but not semantically useful: idle frames, intros, cursor movement, or non-GUI content. In a human verification study over 4,500 frames, the filter achieves 96.0% precision, 91.6% recall, and 93.7% F1 for invalid-frame filtering. The strongest performance is on non-GUI frames, with 98.8% recall. Idle no-action frames are harder, with 53.6% recall, because the interface may look valid while nothing task-progressing happens.

That result is not a side decoration. It explains why naive video mining is dangerous. Pixel change is not action. Visual density is not usefulness. A screen can be full of UI elements and still contribute nothing to the task trajectory. GUIDE’s filter is a quality-control layer before procedural knowledge is synthesized.

The split into planning and grounding is the real product design

After annotation, GUIDE decomposes the video-derived trajectory into two outputs.

Planning knowledge contains the workflow: execution flow, stage objectives, key considerations, and coordinate-free abstractions. It tells the agent what kind of operation sequence the task likely requires.

Grounding knowledge contains UI-element descriptions: names, appearance, screen-relative position, and predicted function for up to a fixed number of key interactive elements. It helps the agent identify the right target in the current screenshot.

This split is not just tidy taxonomy. It controls how GUIDE is integrated.

For a multi-agent system such as AgentS3, planning knowledge is routed to the worker agent responsible for task decomposition, while grounding knowledge is routed to the grounding agent that resolves element descriptions into coordinates. For single-model agents such as Seed-1.8 or Qwen3-VL-8B, both streams are inserted into a unified system prompt, with a structured thought template that asks the model to compare the current task and screenshot against the retrieved reference.

The knowledge is framed as reference material, not as an unconditional command. That matters. Tutorials may use different software versions. They may show a browser when the benchmark environment starts directly inside GIMP. They may solve a neighboring problem rather than the exact task. GUIDE therefore asks the agent to verify the reference against the live screenshot.

This is an understated but important deployment principle: external knowledge should be advisory unless its source is fully controlled. In enterprise settings, that means the agent should treat retrieved SOP videos as strong hints, not divine law. An automation system that cannot say “this reference does not match my current UI” is less an agent than a very expensive intern with confidence issues.

The main evidence says planning is the larger bottleneck

The paper evaluates GUIDE on OSWorld, using 361 tasks after excluding 8 Google-Drive-dependent tasks. The benchmark spans 10 application domains, including Chrome, GIMP, LibreOffice Calc, Impress, Writer, OS operations, Thunderbird, VLC, VS Code, and cross-application workflows.

The headline results are consistent across three agent setups:

The first interpretation is simple: GUIDE improves performance without fine-tuning. The more useful interpretation is narrower: most of the measured gain comes from planning knowledge.

For Seed-1.8, planning alone contributes 6.79 points out of the total 7.48-point gain, roughly 91% of the improvement. For Qwen3-VL-8B, planning contributes 5.03 out of 5.83 points, about 86%. Grounding still helps, especially in UI-complex domains, but the main bottleneck is the agent’s missing knowledge of the software workflow.

This corrects a convenient but incomplete story. GUI agents do not only need sharper vision. They need domain-specific operating knowledge. The agent must know not only that a slider exists, but which dialog should contain it, how to reach that dialog, and which menu convention the application follows.

The per-domain gains sharpen this point. On Seed-1.8, the full planning-plus-grounding configuration improves Writer by 21.74 points and Calc by 19.15 points. These are not domains where the concept of “text” or “spreadsheet” is alien. They are domains where local workflow conventions matter. Office software is a fossil record of product decisions. Agents, like humans, must learn the strata.

The paper also compares GUIDE with Watch & Learn under a broadly comparable inference-time in-context learning setting. Watch & Learn reports a 2.2-point improvement on Jedi, while GUIDE shows a 4.47-point gain on AgentS3. This comparison should not be over-read as a controlled head-to-head between identical systems, but it supports the authors’ claim that live retrieval plus structured dual-channel injection can produce meaningful gains on a strong baseline.

The ablations locate the operating range, not just the applause line

The ablation studies are especially useful because they answer two practical questions: how much grounding information should be injected, and how expensive or capable the annotator needs to be.

The grounding-element-count study uses a 50-task subset with complete grounding annotations. Planning-only scores 41.82. Adding one grounding element raises the score to 45.00. Three elements reach 49.69. Five elements peak at 53.81. Then performance drops: seven elements score 47.88, while ten and fifteen elements plateau at 48.66.

That is an information-noise curve, not a “more context is always better” curve.

For deployment, this is a very practical result. Grounding knowledge helps when it points attention to a small set of discriminative UI elements. It hurts when it becomes a catalog. Anyone who has watched an enterprise prompt grow into a compliance binder will recognize the pattern. Context is not free simply because the model accepts it. Attention is a scarce budget with better branding.

The annotation-model ablation fixes the executing agent and swaps the model used to annotate videos. All tested annotators improve over no annotation. The weakest annotator still adds 15.7 points on the subset, while stronger annotators produce larger gains. The paper reports GPT-4.1 Mini at 47.7, Qwen3-VL-8B at 51.8, GPT-5.1 at 53.8, and Seed-1.8 at 57.8, compared with a 32.0 no-annotation baseline.

The likely purpose of this test is not to crown a universal best annotator. It is a robustness and cost-sensitivity check. The important result is that the structured pipeline contributes a large part of the value. Even weaker annotators, when given subtitles, UI-element graphs, keyframe pairs, and constrained prompts, produce useful knowledge. This matters because a business implementation may not want to pay premium-model costs for every internal training video.

The appendix cost analysis supports the same reading. Under the paper’s assumptions, retrieval costs about $0.0188 per task, while annotation dominates at about $0.252 per selected video using GPT-5.1 in the typical regime. For the OSWorld-scale run, the paper estimates total API spend at $114.6, with annotation accounting for 94.1% of that cost.

This does not mean GUIDE is automatically cheap in every deployment. It means the cost center is legible. If an organization wants to optimize the pipeline, it should not obsess over the search query. It should look at frame-pair annotation volume, UI-parser verbosity, selected-video count, and whether a lower-cost annotator is sufficient.

The qualitative examples show why “knowledge” must be checked against the live screen

The paper’s GIMP example is a clean demonstration of the planning-grounding split. In a contrast-adjustment task, planning knowledge tells the agent that GIMP places brightness and contrast controls under \ast\astColors\ast\ast, not under \ast\astImage\ast\ast. Grounding knowledge then describes the contrast slider as a horizontal bar labeled “Contrast” beneath the brightness slider. Planning gets the agent to the right dialog. Grounding helps it manipulate the right control.

That example is persuasive because it is ordinary. There is no grand reasoning puzzle. No mathematical theorem. No deep symbolic planning. Just a menu convention. This is exactly why GUI automation is hard in practice. Real work fails on dull local details.

The failure cases are equally important. GUIDE can mislead the agent when retrieval selects a procedurally adjacent but wrong tutorial. One GIMP task asks for enhancing a low-resolution photo without increasing file size, but the retrieved video concerns print-resolution settings in PPI. The agent follows the wrong conceptual path and wastes steps. Another case involves converting RAW to JPEG, where the retrieved video discusses installing RAW-processing plugins rather than performing direct export. Again, the planning reference pulls the agent into the wrong workflow.

Grounding can fail too. A CMYK-related tutorial includes browser-based steps for downloading profiles, so the extracted grounding knowledge describes browser UI elements that do not exist in the actual GIMP task environment. In another case, the retrieved “tutorial” is an animated slide deck rather than a real software screencast, so the grounding elements are presentation artifacts instead of GUI controls.

These failures are not minor caveats to sprinkle politely at the end. They define the boundary of the method. GUIDE’s value depends on retrieval quality. The more the retrieved video differs procedurally from the task or visually from the execution environment, the more likely injected knowledge becomes harmful.

That gives us a clean business rule: retrieval confidence is not enough. Procedural compatibility must be tested. A tutorial can be topically similar and operationally wrong. “How to improve image quality” may sound relevant until it teaches print PPI instead of visual enhancement. Semantic adjacency is not procedural equivalence.

The business value is cheaper adaptation, not automatic autonomy

For enterprise AI, GUIDE points toward a practical architecture: a video-to-procedure layer that sits beside the GUI agent.

Instead of fine-tuning an agent for every internal application, a company could retrieve from controlled internal materials: onboarding videos, SOP walkthroughs, support recordings, QA demos, and product training sessions. The system would extract planning knowledge and grounding hints at inference time or precompute them for known workflows. The base GUI agent remains general, while the video-derived layer supplies local software knowledge.

What the paper directly shows is narrower: on OSWorld, GUIDE improves three agent configurations through training-free video retrieval and annotation. What Cognaptus infers for business use is that enterprise automation teams should treat procedural media as a knowledge source, not merely as human training material.

That inference is attractive for three reasons.

First, procedural videos are already abundant. Many organizations have years of screen recordings nobody wants to watch again. This is not a data lake. It is a data swamp with narration. Still, GUIDE suggests there may be extractable operational structure inside it.

Second, the method matches how software changes. Fine-tuned trajectories can become stale when an interface changes. Retrieval from newer tutorials or updated SOP recordings can, at least in principle, refresh the agent’s local knowledge without retraining the base model.

Third, the planning-grounding split maps well to enterprise control. Planning knowledge can be reviewed by process owners. Grounding knowledge can be validated against screenshots. Retrieval filters can be restricted to approved sources. This makes the approach more governable than letting an agent freely improvise with public web content.

But the uncertain part should stay visible. GUIDE does not prove that every enterprise video library is usable. It does not prove safe operation in high-stakes systems. It does not eliminate the need for environment checks, permissions, rollback, logging, or human review. It also does not solve the broader problem of ambiguous business intent. If the task itself is poorly specified, a better tutorial may simply help the agent do the wrong thing more efficiently. That is not progress. That is automation with posture.

A practical deployment framework for GUIDE-like systems

A business team considering this approach should not begin with “Can we connect our agent to YouTube?” That is the fun question, and therefore probably the wrong first question.

A better implementation framework is:

The strongest near-term use case is not full autonomous enterprise operation. It is assisted automation for repeatable software workflows where training videos or SOP recordings already exist and where failures are recoverable. Examples include data-entry workflows, report formatting, document preparation, low-risk CRM updates, internal admin tooling, and software QA assistance.

The weaker use case is open-ended business process automation across poorly documented systems with high consequences. GUIDE-style retrieval can help an agent find its way around software. It cannot decide whether the business process itself should exist. For that, unfortunately, someone still has to do management.

The appendix tests robustness, cost, and failure boundaries rather than a second thesis

The appendix material is worth reading because it explains how much of the system is engineering discipline rather than benchmark theater.

The prompt-structure appendix shows how planning and grounding knowledge are inserted differently for multi-agent and single-model systems. This supports the architecture-agnostic claim: the knowledge format is natural language, so integration does not require weight updates. But it also reveals a dependency: prompt design is part of the product. GUIDE works by making the agent actively consult the reference, not by passively appending a long note and hoping for spiritual absorption.

The benchmark details clarify that OSWorld tasks run in live Ubuntu virtual machines with execution-based evaluation and a maximum of 50 interaction steps. This matters because the agent must actually change system state, not merely describe what it would do.

The cost appendix identifies annotation as the dominant cost. That is not a limitation by itself; it is a lever. If the business case requires scale, the engineering problem becomes reducing selected-video count, shortening annotation trajectories, compressing UI-element descriptions, caching reusable knowledge, and selecting cheaper annotators where quality remains acceptable.

The failure-case appendix is perhaps the most practically honest part of the paper. It shows that retrieval mistakes are not harmless. A wrong video can become a wrong plan. A non-screencast can become useless grounding. These are not generic “AI may fail” warnings. They are specific failure modes with specific mitigations: stricter procedural filtering and video-format classifiers.

That specificity is what makes the paper useful. A limitation that cannot guide design is just an apology in academic clothing.

The agent does not need a better eye as much as a better apprenticeship

GUIDE’s best idea is not that GUI agents should watch YouTube. That framing is catchy, but too shallow. The better idea is that software operation is learned through procedural examples, and those examples can be converted into structured, task-time knowledge without retraining the whole agent.

For AI automation, that shifts the design conversation. Instead of asking only whether a model can perceive a screen and emit an action, we should ask what apprenticeship materials surround the agent. What tutorials, SOPs, demos, and recorded workflows can it consult? How are those materials filtered? How are they converted into planning and grounding knowledge? How does the agent verify them against the current environment?

The paper’s evidence suggests that planning knowledge is the larger lever, grounding is complementary, and retrieval quality is the central boundary. That is a useful map. Not a complete product. Not magic. But a map.

And for enterprise automation, a map is already better than asking a generalist model to wander through LibreOffice like a tourist with admin privileges.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast