Train the Agent Before the Sandbox Exists

TL;DR for operators

A team can have stable API contracts before it has an executable sandbox, populated user state, and reliable integration infrastructure. The usual assumption is that serious multi-step agent training must wait, because later API responses need to reflect what earlier calls changed.

ESAT challenges that sequencing. Lee et al. generate stateful training trajectories from API specifications without the target environment, then validate and filter the interactions before training.¹ On AppWorld, training with ESAT-S52 trajectories generated from 52 synthetic applications independent of AppWorld improves Task Goal Completion by 8.4 to 47.0 percentage points across evaluated models and usually outperforms training on trajectories collected inside the real AppWorld environment.

For API teams, the decision is therefore not whether a realistic sandbox matters, but whether it must exist before training starts. The evidence supports moving part of training upstream to the API-contract stage while reserving real execution for validation, edge cases, deployment testing, and production confidence.

The expensive prerequisite may be the environment, not the API contract

Generating useful supervision for a multi-step API agent is harder than writing plausible tool calls. Suppose an agent creates a calendar event, updates it, then searches for events matching the new state. The third response must reflect what happened in the first two calls. A collection of independently fabricated API responses will not teach that interaction reliably.

This is why environment construction becomes a bottleneck. Executable APIs need backend logic. Stateful tasks need realistic databases. Cross-application workflows need several services to behave coherently at once. Before training even begins, a team may have to reproduce much of the integration environment it eventually wants the agent to operate.

ESAT changes what is required at that stage. Instead of calling a real backend, a teacher agent issues API requests step by step while a separate LLM generates the responses. That simulator sees the API specification, call arguments, user task, previous interactions with the same application, a virtual-user profile, and the current time. Earlier simulated writes therefore become part of the context used to produce later reads.

The useful abstraction is not a fake endpoint. It is a constrained digital world that persists long enough to generate a training trajectory.

State consistency is enforced rather than assumed

The simulator is only one part of the pipeline because plausible language is not sufficient evidence that an API interaction is valid.

Each generated response first passes deterministic schema checking. A second LLM-based check evaluates semantic plausibility and consistency with prior state. Failed checks trigger structured retries. Once the agent completes the task, a separate judge evaluates the trajectory as a whole before it becomes supervised fine-tuning data.

Task generation is also designed around coverage. ESAT varies task difficulty, read-versus-write behavior, number of applications, task focus, and API-count ranges. Underused applications receive greater sampling weight, while individual APIs can be deliberately spotlighted. In the retained AppWorld-derived dataset, 339 of 340 APIs appear in at least one successful trajectory.

That coverage mechanism matters operationally because synthetic pipelines can otherwise become self-reinforcing: endpoints that are easy for the generator appear frequently, which gives the student more practice on exactly the APIs it already sees most often.

Synthetic supervision transfers beyond the synthetic environment

The central experiment is not whether ESAT can imitate AppWorld after seeing AppWorld state. ESAT-S52 is generated from 52 LLM-designed synthetic applications covering 1,017 APIs, independently of AppWorld. Fine-tuning on those trajectories nevertheless raises AppWorld Task Goal Completion by 8.4 to 47.0 percentage points across the evaluated models and usually outperforms AppWorld real-environment training trajectories.

That result makes the paper more consequential than a cheaper data-generation demonstration. The student appears to learn transferable interaction behavior rather than only memorizing the states present in the synthetic generator.

Adding trajectories generated from AppWorld API specifications improves results further. Qwen3.5-9B trained on the combined ESAT-S52-AW7 data reaches 75.7 TGC on AppWorld Test-N and 65.7 on Test-C, roughly 50 points above its zero-shot baseline. Qwen3.5-27B reaches 84.2 and 79.0, within three points of the much larger GLM-5.1-FP8 teacher.

OfficeBench provides a second environment. There, ESAT training improves performance by 5.2 to 60.5 percentage points over zero-shot results. Qwen3.5-2B rises from 9.1 to 69.6 on two-application tasks and from 0.7 to 40.9 on three-application tasks.

These are benchmark comparisons rather than causal estimates, but the repeated pattern across eight student models makes the environment-free training result difficult to dismiss as a single-model artifact.

Filtering is part of the training architecture

The paper’s trajectory-filter comparison is best read as an ablation: it asks whether curation quality changes what the student learns.

The more selective Gemini judge keeps fewer trajectories but produces the strongest downstream Qwen3-8B result. More synthetic data is therefore not automatically better training data.

The authors validate that judge against evidence outside the judge itself. On 720 AppWorld trajectories labeled through executable verifiers, it achieves 95.2% precision. On a separate 200-trajectory human review sample, agreement reaches 95%, with Cohen’s kappa of 0.90.

For teams building synthetic-data pipelines, this suggests a concrete control structure: validate individual responses mechanically where possible, inspect semantics and state transitions, then evaluate complete trajectories before training. Generation and quality governance are one system.

What this changes for API teams

The business interpretation is mostly about sequencing.

The largest potential saving is therefore not token cost. It is reducing how much backend infrastructure must exist before an organization can begin producing agent-training data.

Long responses expose the reliability boundary

The simulator analysis is a reliability test, not a second performance claim. Across about 27,000 simulated API calls from 1,000 successful trajectories, 93.7% are judged valid. That is high enough to explain why useful training data can emerge after validation and filtering, but not high enough to treat simulated state as ground truth.

Output length is the clearest failure signal reported. Failure is about 2–3% below 500 tokens, around 8% from 500–1,000, 18% from 1,000–2,000, and 23% above 2,000 tokens.

An organization considering the approach should therefore classify APIs by response complexity. Short structured reads may be relatively straightforward simulation targets. APIs returning long documents, large collections, or complex nested state deserve tighter validation, chunking, executable spot checks, or direct environment generation.

There is also meaningful attrition before data reaches training. ESAT retains 65.13% of input AppWorld-derived tasks, 59.62% for the synthetic-app dataset, and 49.38% for OfficeBench. The pipeline works partly because it discards substantial amounts of generated material.

Training can move upstream; confidence cannot

The paper changes a useful engineering assumption. Executable environments are not necessarily prerequisites for generating effective multi-step API-agent supervision. With state-aware simulation, coverage management, response validation, and trajectory filtering, an API specification can become a training asset much earlier in the product lifecycle.

What remains uncertain is how far this transfers beyond the two evaluated benchmarks, especially for proprietary APIs with unusual state semantics, large outputs, irreversible actions, or safety-critical consequences.

The resulting workflow is therefore not “replace the environment with an LLM.” It is more disciplined: use specifications and simulation to start training earlier, then spend real-environment capacity where simulation provides the least confidence—validation, edge cases, and deployment-critical behavior.

Cognaptus: Automate the Present, Incubate the Future.