When a Failed Workflow Still Produces a Learning Signal

TL;DR for operators

An automation agent may retrieve the right information, update one application, and complete several intermediate operations before failing its final task. A binary success test assigns that entire attempt zero reward, even when parts of the workflow were executed correctly.

Explorer-Zero¹ demonstrates a training alternative: let one agent generate achievable tasks through actual environment interaction, record verifiable intermediate states, and let another agent learn from those checkpoints.

In an AppWorld component experiment, adding intermediate supervision increases task goal completion from 45.83% to 59.67%, a gain of 13.84 percentage points. The benefit is particularly pronounced on complex tasks.

For organizations developing workflow agents, the practical opportunity is to create training environments where partial achievements can be checked automatically. The boundary is equally consequential: the method requires substantial environment infrastructure, and verified progress does not establish that an action is safe, authorized, or representative of customer needs.

A failed task can contain several different learning outcomes

Consider an agent responsible for a workflow spanning multiple business applications. It retrieves a record, creates a related task, and prepares an update in another system. The final operation fails.

A conventional task-level evaluator reports failure. During reinforcement learning, another attempt that never retrieved the record might receive exactly the same score.

These trajectories are not equally informative. One has demonstrated several capabilities needed to complete the workflow; the other has not. Yet when both receive zero reward, the training system loses that distinction.

The problem becomes especially consequential when tasks are difficult enough that nearly every sampled attempt fails. Group relative policy optimization (GRPO), the training algorithm used in Explorer-Zero, learns from differences among sampled outcomes. If every attempt receives zero, there is no comparative reward advantage for that group.

The central question is therefore whether intermediate achievements can be rewarded reliably without redefining an unfinished task as successful.

The proposer generates both the task and its verification checkpoints

Explorer-Zero uses two separately trained roles.

The proposer explores an interactive environment, identifies a task, performs the operations needed to demonstrate it, and records relevant intermediate and final environment states. It produces a natural-language instruction supported by those observed states.

The solver receives the proposed task and attempts it independently. It does not receive the proposer’s demonstration trajectory. Its actions are evaluated against the recorded reference conditions.

This arrangement tackles two training requirements together: finding achievable tasks and constructing checks for their completion.

A proposer that has actually completed a workflow provides stronger evidence of feasibility than a model that merely writes a plausible instruction. Its recorded states can also provide verification targets without separately authoring a task-specific test for every generated example.

The proposer is rewarded for generating suitably difficult tasks, nonredundant subtasks, and diverse action types. Difficulty is estimated by testing the current solver: tasks completed approximately half the time receive the highest difficulty reward, while nearly impossible or trivial tasks receive little.

Alternating proposer and solver training lets the generated curriculum change as the solver improves.

There is an important infrastructure requirement behind this design. The environment must expose sufficient state to record and compare task-relevant outcomes. A demonstrated sequence alone does not guarantee that every natural-language requirement has been captured by the verifier.

Partial rewards work when failures differ

Explorer-Zero gives a successful solver trajectory full reward. If the overall task fails, its reward is instead the fraction of declared subtasks that were verified during execution.

For example, when a task contains four checkpoints, an unsuccessful attempt that reaches three receives 0.75. Another that reaches one receives 0.25. Both remain unsuccessful according to the final-task criterion.

Checkpoint verification can consider any state visited during the solver trajectory, not only the terminal state. This allows an intermediate achievement to count even if its effects are later modified.

The paper’s Proposition 1 explains the optimization consequence.

With binary rewards, a group in which every sampled trajectory fails always receives identical zero scores. Under decomposable rewards, that group can retain reward variation if different attempts complete different numbers of subtasks.

Under independent, identically distributed rollouts, the conditional probability of identical subtask rewards is strictly below one when at least two different completion counts have positive probability.

This is a specific conditional result, not a guarantee that partial rewards always improve learning. If every failed attempt completes the same number of checkpoints, the reward differences still disappear.

The reward also has a semantic limitation. Subtasks form an unordered verification checklist, not an explicitly modeled causal dependency graph. An unsuccessful attempt can even receive a reward of one if it completes every declared checkpoint without satisfying the final task.

Intermediate supervision therefore preserves potentially informative distinctions, but the quality of those distinctions depends on checkpoint design.

The ablation separates task generation from partial-progress supervision

The component ablation is particularly informative because it distinguishes the contributions of interactive task generation and intermediate rewards.

Source: Section 4.2, Table 2. AppWorld uses Qwen3-30B-A3B-Instruct-2507. The corresponding ALFWorld ablation rises from 65.71% to 84.29% when subtask supervision is added, using the valid-seen split.

Interactive proposal improves on one-shot proposal. However, the one-shot configuration also uses a different method of obtaining reference outcomes, so this comparison does not isolate interaction alone.

The cleaner comparison is between the two interactive configurations. Both generate tasks through environment interaction; the full method additionally supplies intermediate supervision. The resulting gains of 18.22 points on ScienceWorld and 13.84 points on AppWorld indicate that the contribution extends beyond generating executable tasks.

The broader benchmark results reinforce this finding. Explorer-Zero achieves 85.07% success on ALFWorld’s unseen validation split, compared with 66.40% for Self-Challenging Agents. On ScienceWorld, the comparison is 42.92% against 35.69%. On AppWorld, the 30B-A3B model reaches 59.67% task goal completion against 51.93%.

These are comparative results within controlled benchmarks, reported over four sampled evaluations per task prompt. They are not estimates of production deployment effectiveness.

The benefit grows with workflow complexity

Additional analyses examine whether subtask rewards encourage meaningful task composition or simply longer sequences of actions.

On AppWorld, dependency-chained segments per generated proposal increase from 0.57 to 1.13 across the compared training blocks. Environment-grounded subgoals rise from 0.55 to 1.22, while verified segments increase from 0.29 to 0.77.

The authors use program slicing and API data-flow analysis to examine relationships among the generated operations. These are mechanism and structural analyses, not a second independent benchmark of task success.

A revealing qualification is that distinct APIs per task and the proportion of multi-application tasks decrease during the same analysis. More compositional structure therefore does not necessarily mean broader exploration.

The complexity-stratified results provide another clue: subtask supervision produces relatively limited consistent gains on simpler AppWorld tasks, with substantially larger benefits in the highest-complexity groups.

For enterprise-agent training, this suggests that intermediate checkpoints may be most valuable where several dependent operations must succeed before a final outcome can be achieved.

Self-play can complement curated training data

Explorer-Zero approaches, but does not exceed, reinforcement learning on the full human-annotated training set in the reported AppWorld 30B-A3B task-completion comparison: 59.67% versus 62.65%.

The more consequential result for organizations already maintaining curated tasks is that self-play can improve subsequent training.

On ScienceWorld, human-data reinforcement learning achieves 49.63% success; Explorer-Zero initialization followed by that training reaches 63.26%. The corresponding AppWorld result increases from 62.65% to 69.20%.

This supports investigating self-generated curricula as a preparation stage that complements curated task data, rather than assuming human-authored examples should be eliminated.

Controlled training environments remain the boundary

The method depends on resettable, observable environments with comparable reference states. Live business applications may contain irreversible operations, hidden state, inconsistent external effects, and permissions that cannot be inferred from outcome matching.

It also has measurable costs. The reported experiments use an eight-H100 GPU node, with approximately 29 node-hours for an AppWorld training run. Eliminating human-annotated training tasks does not eliminate rollout computation or verification infrastructure.

A practical investigation should therefore measure final-task success, checkpoint validity, within-group reward variation, training cost, and coverage of representative business workflows. Generated tasks must also be assessed for relevance to the operations an agent is actually expected to perform.

Authorization and safety require independent controls. A successful state transition does not imply that the agent was permitted to produce it.

The next training investment is better-verified progress

Explorer-Zero identifies a specific weakness in long-horizon reinforcement learning: unsuccessful trajectories can contain meaningful differences that final-outcome rewards erase.

Its contribution combines interaction-grounded task generation, reusable state checkpoints, and a reward mechanism that preserves those differences under identifiable conditions. The benchmark gains suggest that this combination is particularly relevant to complex workflows.

For enterprise developers with resettable software test environments, the next experiment is to measure whether intermediate verification produces better-trained agents at an acceptable infrastructure and computation cost. The result should be evaluated against real task requirements and final outcomes, not checkpoint completion alone.

Verified progress can improve how an agent learns from failure. It is not, by itself, evidence that the agent is ready to act in production.

Cognaptus: Automate the Present, Incubate the Future.