The Right Tool Is Not the Whole Workflow

TL;DR for operators

An AI agent can select the correct function and still fail the overall workflow because an earlier result was lost, a prerequisite was skipped, or the wrong patient or treatment-plan identifier was passed downstream.

RadOnc-Agent makes these risks measurable. In matched synthetic tests, preserving longitudinal workflow state increased completion from 84.00% to 96.50%. A separate validation experiment found that deterministic checks prevented all deliberately mismatched backend dispatches, whereas bypassing those checks allowed 95.28% to proceed through restricted replay/test interfaces.

For organizations building or purchasing multi-system AI agents, the implication is architectural: persistent state, structured information transfer, and enforceable validation deserve their own design requirements and evaluation metrics. High tool-selection accuracy is insufficient evidence of reliable end-to-end execution.

The research demonstrates technical feasibility in radiotherapy, including retrospective patient-specific workflows. It does not establish that the system makes clinically correct decisions or improves patient outcomes.

A workflow can fail even when its individual tools work

Consider a radiotherapy process in which one system identifies the treatment intent, another prepares a planning directive, and a third evaluates whether anatomical changes require reconsidering the treatment plan.

Each function might perform its assigned task correctly. The process can nevertheless fail if the planning directive loses its prescription context, a subsequent computation uses the wrong reference plan, or an adaptation assessment runs before the required imaging comparison is available.

These failures concern the connections between tasks rather than the competence of any single model.

In RadOnc-Agent, Jiang and colleagues¹ investigate this coordination problem through an LLM-orchestrated framework spanning diagnosis and decision-making, simulation and planning, on-treatment adaptation, and follow-up.

The system provides access to 26 capabilities: 17 phase-specific functions and nine cross-cutting utilities. A GPT-5.2 controller interprets requests and coordinates independently implemented specialist services, while a shared function registry specifies their inputs, outputs, and execution requirements.

The architecture’s distinctive contribution is not a single model performing every clinical calculation. It is the controlled movement of information between specialized functions over successive stages of care.

Accurate tool selection leaves another failure surface

The system performed strongly when asked to select individual functions.

Across 2,600 unique requests, each executed three times, intended-function selection reached 98.79%. A matched comparison against an embedding-based router yielded 95.49% for the baseline. The LLM controller’s advantage was especially pronounced for semantically confusable requests, where it led by 7.31 percentage points.

That comparison establishes a benefit for language-based routing within the tested function registry. It does not test whether the selected functions can complete a long clinical workflow together.

The synthetic cross-stage evaluation makes that additional requirement visible. Overall workflow completion reached 96.50%, but full-pathway workflows completed at 92.67%, below the 96.67%–98.67% rates reported for shorter workflow families.

These percentages concern different evaluation tasks and should not be interpreted as a direct conversion from tool-selection accuracy to clinical success.

The failure analysis is more informative. Among 21 incomplete synthetic executions, nine involved unsuccessful propagation of upstream outputs and six involved prerequisite or ordering violations. Together, these accounted for 15 failures. The remainder involved persistent backend failures or unresolved missing inputs.

A correct next-function choice does not guarantee that the function receives the necessary evidence, identifiers, and previously generated outputs.

Persistent state changes whether the workflow finishes

The strongest architectural result comes from removing persistent workflow state while keeping the controller model, function registry, validation layer, and evaluation scenarios fixed.

Source: Extended Data Table 2; 200 unique synthetic cross-stage scenarios, each executed three times.

The 12.50-percentage-point completion improvement is substantial under these matched benchmark conditions. Importantly, the information-transfer difference provides a mechanism consistent with that improvement.

Persistent state retains structured outputs, patient and treatment-course identifiers, earlier decisions, and execution provenance. A downstream function can then consume an explicitly retained result rather than reconstructing its context from the latest instruction.

Reference-trajectory adherence measures something additional: whether the required actions follow the specified sequence and dependencies. A workflow can produce a final output while still departing from its prescribed execution path.

This evidence suggests that operational memory should be treated as an independently testable system component. Conversational context alone is not an adequate substitute for structured workflow records when later actions depend on precisely identified earlier outputs.

The ablation supports that conclusion within the tested architecture. It does not establish that the same performance gain would necessarily occur with every controller model, institution, or workflow configuration.

Validation must control execution, not merely advise the model

Preserving the correct state solves only part of the problem. An agent may still propose a call with missing inputs or inconsistent identifiers.

RadOnc-Agent separates the controller’s proposed action from the system’s decision to dispatch it. Deterministic checks examine function contracts, required arguments, and patient, course, and plan identifiers before backend execution.

The missing-input benchmark illustrates why this separation matters. Across 1,560 executions, the controller produced safe clarification or withholding in 97.18% of cases. In the remaining 2.82%, it attempted to proceed inappropriately before enforcement by the validation layer.

A more direct architecture ablation deliberately injected identifier mismatches:

• Validation enabled: zero mismatched dispatches across 360 replay/test executions.
• Validation bypassed: 343 of 360 mismatched calls were dispatched, or 95.28%.

No bypassed call was sent to an actual registered clinical service.

This experiment isolates a technical safeguard under deliberately constructed invalid-input conditions. It does not estimate how frequently such mismatches arise in clinical practice.

Nevertheless, it establishes a significant engineering distinction. An LLM’s judgment about which action to propose is separate from the system’s enforcement of whether that action has valid inputs and consistent identity information.

The checks do not themselves establish clinical appropriateness or authorization for routine clinical use.

Retrospective execution establishes a narrower form of feasibility

The authors also evaluated RadOnc-Agent against linked records from 60 de-identified Mayo Clinic Arizona patients, producing 120 patient-specific workflows and 360 clean repeated executions.

Workflow completion reached 96.67%, with 95.56% adherence to the required execution trajectories. Patient, course, and plan identifier propagation succeeded in 351 of 354 eligible hand-offs, with no observed cross-patient contamination.

These results matter because the framework operated on linked clinical objects rather than exclusively synthetic scenarios.

The scope, however, remains specific. Retrospective real-patient testing covered decision-to-planning and planning-to-adaptation transitions, not complete four-phase care pathways. The cohort came from one institution and required sufficiently complete, linkable records.

The study also reported 91.67% agreement between replan-assessment outputs and subsequently recorded treatment-plan changes across 60 planning-to-adaptation workflows.

That agreement is not a clinical correctness rate. A recorded plan change establishes what happened in subsequent care, not whether changing the plan was medically necessary or whether an unchanged plan was optimal.

Separate perturbation tests assessed recovery from missing inputs and transient backend failures. Recovery reached 84 of 90 synthetic executions and 45 of 48 retrospective patient-specific stress-test executions. These are useful technical resilience checks, not evidence of prospective operational safety.

No prospective assessment established clinician time savings, editing burden, treatment quality, or patient benefit.

What enterprise agent buyers should test separately

The Cognaptus inference extends beyond radiotherapy, with an explicit boundary: the paper evaluates a specialized clinical integration architecture, not enterprise workflows in general.

For an organization coordinating consequential processes across multiple applications, the study suggests several procurement and engineering questions.

Can the agent preserve the identity and provenance of intermediate results? Can it demonstrate that each downstream function receives the intended upstream output? Does it enforce prerequisites before execution? Are invalid calls rejected mechanically, even when the language model proposes them? Can it resume after a failed service without losing the workflow’s state?

These capabilities should be tested with multi-step scenarios, deliberate missing-input and identifier errors, and failures injected at different stages. Reporting only whether the agent selected the correct tool would leave much of the operational risk unmeasured.

A useful evaluation therefore distinguishes routing accuracy, workflow completion, correct execution order, information-transfer accuracy, validation behavior, and recovery performance.

For healthcare buyers, these tests must remain separate from specialist-model clinical validation, institutional permissions, and prospective assessment of clinician decisions.

The paper does not show that the architecture reduces costs or transfers successfully across institutions. Those remain empirical questions for deployment evaluation.

The reliability question moves between the calls

RadOnc-Agent demonstrates that the reliability of an integrated AI workflow depends materially on what happens between specialized computations.

The matched experiments identify persistent state as a contributor to workflow completion and deterministic validation as a barrier against inconsistent backend execution. The retrospective evaluation shows that selected cross-stage workflows can be executed using linked patient records.

For operational AI design, the resulting priority is clear: evaluate the continuity, dependencies, and execution controls of the whole process alongside the abilities of its individual models.

Clinical effectiveness remains unproven. But the study provides a concrete way to investigate a class of failures that accurate tool selection alone cannot resolve.

Cognaptus: Automate the Present, Incubate the Future.