The Agent Needs the Cluster, Not Just the Documentation

TL;DR for operators

European XFEL did not find that the main obstacle to scientific AI assistance was simply insufficient documentation. Scientists and support staff had to combine project objectives, facility knowledge, instrument procedures, specialized software, computing constraints, and expertise distributed across people and documents.

The resulting study¹ points toward a different implementation model. An effective scientific agent needs grounded retrieval, but it also needs access to the actual execution environment, mechanisms for testing generated code, visible sources and plans, approval gates for sensitive actions, and components that can be replaced as models and tools change.

The first prototype makes the distinction concrete. It could retrieve facility knowledge inside a notebook, yet still produced incorrect library calls because relevant context was incomplete and the system had no agent loop for executing, checking, and repairing its work.

For organizations building internal agents, this shifts architecture discussions away from selecting the strongest model first. The harder questions concern what the agent can know, what it can execute, what must be verified, which actions require approval, and how cheaply each component can later be replaced.

Scattered documentation is only one part of the overload

Imagine a scientist with a computational notebook already open. The next analysis step may depend on a facility-specific library, an instrument procedure, the structure of the current project, an HPC constraint, and an undocumented convention that an experienced colleague normally supplies.

European XFEL found this coordination problem across several stakeholder groups. Among the 13 challenges identified in the study, both facility users and Data Analysis members reported difficulty retrieving and synthesizing scattered documentation. Users also faced unfamiliar infrastructure, HPC systems, and instrument-specific procedures. Support staff, meanwhile, spent time reconstructing users’ objectives and supplying missing expertise.

That evidence changes what “knowledge support” means. The problem is not just finding a relevant paragraph. The system must assemble enough context to move a particular analysis forward.

The authors translated those observations into 21 requirements spanning architecture, agent behavior, retrieval, code generation, and optional support functions. The list includes access to the HPC cluster, project context, verifiable goals, citations, code testing, user feedback, and approval for sensitive actions.

This is closer to workflow infrastructure than an improved search box.

Retrieval failed exactly where execution began

The first prototype already had several ingredients that are often presented as the answer: notebook integration, an internal retrieval backend, a language model, embeddings, and remote access to a Jupyter kernel on the HPC cluster.

Yet retrieval-grounded assistance still generated incorrect library calls.

The likely purpose of this prototype evaluation was not to benchmark retrieval quality against alternative systems. It exposed a design failure: retrieved documents did not always contain enough project and workflow context to generate correct facility-specific code, while the prototype lacked an agent loop capable of testing a proposed solution and repairing it.

That distinction matters because retrieval-augmented generation only changes what information reaches the model. It does not establish that the resulting code is executable, appropriate for the current environment, or scientifically correct.

The paper’s architecture therefore moves beyond retrieval toward a sequence in which the agent can obtain context, generate code, operate against the real computational environment, inspect results, and keep the scientist able to review consequential actions.

For an engineering organization, the analogous boundary is straightforward: connecting an assistant to internal documentation does not make it competent to modify a production system. Knowledge access and action reliability are different engineering problems.

The useful unit of autonomy is a controlled action

The second prototype connected an off-the-shelf agentic tool to the Jupyter environment and multiple knowledge and reporting services. In 13 one-hour expert sessions, participants valued retrieval, generated code, notebook execution, responsiveness, and reporting. The system also required repeated configuration and refinement.

The more durable contribution is the control architecture that emerged from this work.

This is bounded autonomy in operational terms. Routine information access need not attract the same approval burden as editing files or executing code on an HPC system.

Cognaptus inference: the same principle can improve enterprise agent governance. Human review becomes more scalable when it is attached to the consequence of an action rather than every turn of an agent conversation.

The study does not quantify the productivity gain from doing this. It provides an architecture for allocating authority, not evidence that the allocation produces a particular return.

Some instructions should not remain instructions

One of the more consequential findings concerns configuration.

The researchers used natural-language files such as AGENTS.md and skill definitions to shape agent behavior. They found that behavior was not stable across models, reasoning settings, and variations in instruction wording.

Their recommendation is therefore stronger than “write better prompts.” Behaviors that must happen reliably should be backed by programmable hooks or equivalent software controls.

This separates two categories that organizations often mix together. Preferences such as response style, preferred libraries, or workflow conventions may remain natural-language configuration. Requirements such as obtaining approval before execution, following a required retrieval order, or checking the execution environment are candidates for programmatic enforcement.

That division turns agent governance into system design rather than prompt maintenance.

Documentation becomes part of the execution stack

The study also reaches an unusually operational conclusion about documentation. API references were often insufficient for library-specific code generation. Agents benefited from documentation that was structured, complete enough to retrieve reliably, and oriented around workflows rather than isolated functions.

This does not make human-readable documentation obsolete. It adds another consumer.

For documentation teams, Cognaptus infers a new infrastructure responsibility: examples, task sequences, assumptions, environment requirements, and relationships among tools can affect the quality of downstream agent behavior. Documentation quality can therefore become an input to execution reliability, not merely support cost.

Build for replacement, not permanence

The study evaluated 16 free or open-source tools and components, but the authors explicitly do not present that exercise as a quantitative benchmark. The selection of tools for the second prototype should not be read as evidence that those products are universally superior.

Instead, the authors recommend modular components that can be independently configured and replaced.

That distinction is particularly relevant in the current agent ecosystem. Minimizing the number of components may simplify an initial prototype while making later migration harder. A modular design accepts some integration complexity in exchange for the ability to replace the model, retrieval service, connectivity layer, reporting system, or agent interface without rebuilding the complete workflow.

For platform and procurement teams, the decision criterion becomes workflow fit and replacement cost: privacy constraints, infrastructure compatibility, latency, interoperability, maintenance effort, and the effort required to switch components.

What this study does not establish

The evidence is design guidance from one specialized scientific facility. Prototype participants were European XFEL staff rather than a representative sample of visiting users. The component comparison was not a benchmark. Production-grade autonomous-action safety was not tested in depth, and the second prototype was not evaluated through a long-term deployment.

Most importantly, the paper does not estimate how much agentic support improves scientific productivity, analysis speed, or scientific validity.

Those boundaries limit claims about economic return and general adoption. They do not remove the architectural lesson the study exposes: once an AI assistant is expected to participate in real work, retrieval quality is only one dependency among several.

The deployment decision starts with the workflow

The 14 final design recommendations cover architecture, approvals, retrieval, documentation, model selection, specification-driven code development, minimal iterative solutions, and explicit treatment of the execution environment.

Taken together, they suggest a useful order for implementation.

Map the workflow knowledge first. Identify the real environments in which actions occur. Decide which actions require verification or approval. Determine which behaviors need hard enforcement. Then select models and components that can satisfy those requirements while remaining replaceable.

For knowledge-intensive organizations, that ordering may be more durable than beginning with a chatbot and progressively granting it more access.

A scientific agent becomes materially different from a retrieval interface when it can cross the boundary between explaining work and participating in it. The system architecture determines whether that crossing is inspectable, constrained, and maintainable.

Cognaptus: Automate the Present, Incubate the Future.