A Test Result Is Not a Regulatory Agreement

TL;DR for operators

An AI compliance platform can execute technical tests correctly without resolving whether the resulting evidence addresses the right regulatory questions. Engineers, assessment specialists, and regulators bring different responsibilities and interpretations to the same system.

A new study develops a collaborative assessment tool that combines AI system qualification, regulatory controls, technical tests, shared dashboards, and reporting. Three design iterations revealed why testing functionality alone was insufficient: some stakeholders struggled to interpret the original technical interface, while others needed assessment pathways that did not begin with executable tests.

The resulting architecture organizes these functions across six phases of an AI regulatory sandbox. For compliance platform designers and participating organizations, the implication is to connect system classification, assessment scope, evidence, and supervisory interpretation within a shared workflow.

This is stakeholder-informed design evidence, not proof of operational effectiveness. The researchers have not demonstrated lower compliance costs, faster assessments, or successful interoperability between national sandboxes.

Technical understanding does not guarantee regulatory understanding

Consider an AI company entering a supervised regulatory assessment. Its engineers understand the system, technical specialists can evaluate model behavior, and regulatory authorities must interpret the results against applicable obligations.

A shared testing platform appears to simplify their work. Yet a technically valid result may not answer the regulator’s question. The participants must first agree on what system is being assessed, which requirements matter, and what the resulting evidence establishes.

This difficulty surfaced during the first design iteration reported by Idoia Landa-Oregi and colleagues in Designing a Boundary Negotiating Artifact for Collaborative Socio-Technical Sense-Making in AI Regulatory Sandboxes.¹ Their initial assessment concept was understandable to technical participants but less accessible to non-technical stakeholders, particularly regulators (Section 4.1).

The observed problem was not a defective test. The interface could communicate technical functionality without providing everyone with an equally usable basis for regulatory interpretation.

For organizations developing assessment platforms, this changes the design question. Adding more tests may improve analytical coverage while leaving unresolved how participants decide which tests are appropriate and what their results mean.

The missing component is negotiated meaning

The European AI Act establishes AI Regulatory Sandboxes (AIRS) as supervised settings for innovation and regulatory learning. Such environments bring together stakeholders who possess different kinds of authority and expertise.

The paper builds on the concept of a Boundary Negotiating Artifact (BNA): a shared representation through which participants develop, challenge, and revise interpretations during collaborative work.

An established boundary object can help different professional communities coordinate around relatively stable meanings. A negotiating artifact supports the earlier process in which those meanings are still being worked out.

In regulatory assessment, this distinction matters because the participants contribute different judgments:

• AI innovators explain system characteristics, intended uses, and implementation details.
• Technical experts translate assessment needs into procedures, tests, and interpretable technical evidence.
• Regulators provide legal interpretation, guidance, and supervisory judgment.

The proposed digital artifact gives these groups common representations to inspect and discuss. Its purpose is not to automate legal judgment or eliminate differences in expertise.

The underlying mechanism is continuity of interpretation: system descriptions inform assessment choices, tests produce evidence, and shared results support further discussion and reporting. Whether this mechanism improves real regulatory decisions remains an empirical question.

Three design iterations expanded the scope beyond testing

The researchers use Research-through-Design, developing successive artifact versions and using stakeholder interactions to identify design requirements. The three iterations provide the paper’s main evidence for its architectural decisions.

The first iteration presented an assessment concept to ten attendees representing the principal stakeholder groups. Feedback identified an accessibility gap between technically oriented participants and others who needed to understand the assessment process.

The second iteration introduced a demonstrator featuring a harmonized test catalog, execution engine, dashboards, and reporting. It was presented informally to approximately 40 conference attendees and examined in a focused workshop involving 60 participants.

This feedback revealed a more fundamental limitation. Executable technical tests were not always the appropriate starting point. Stakeholders needed to qualify the AI system and identify applicable assessment requirements before selecting procedures. Some systems also required regulatory controls that did not necessarily involve technical execution.

The third iteration therefore added system qualification and regulatory control checklists. Feedback from a focused presentation involving 30 attendees and approximately 40 additional informal demonstration discussions was positive about the broader scope and simplified organization of assessment options.

Stakeholders also valued harmonized descriptions and procedures because they reduced the need to understand separate interfaces for different testing tools. This was perceived utility, not a measured reduction in assessment time.

The important result is the direction of the design revisions. Stakeholder feedback moved the artifact from coordinating technical execution toward coordinating decisions about assessment scope, evidence, and regulatory interpretation.

These engagement counts represent separate events, not a documented deduplicated sample. The iterations identify recurring design concerns; they do not constitute controlled effectiveness comparisons.

One collaborative process, six different phases

The resulting architecture combines system qualification, an extensible assessment catalog, regulatory controls, configurable testing through a plug-in interface, collaborative dashboards, and structured reports.

The authors then place these functions within a six-phase sandbox framework. Figure 4 (Section 5, PDF page 10) assigns responsibilities to innovators, regulators, and technical experts while identifying where the tool contributes.

This framework treats the BNA as a coordinated collection of lifecycle-specific interactions. A qualification form supports a different conversation from a testing dashboard, while a final report consolidates information for another decision.

The regulator remains responsible for supervisory and evaluative activities. Technical experts retain assessment responsibilities, and innovators supply system information and participate in testing.

The six phases are a proposed operational synthesis, not a newly established legal procedure. The distinction is relevant when organizations translate this architecture into their own compliance workflows.

What compliance teams can take from the architecture

Cognaptus inference: The paper offers a reference for designing AI assessment systems around decision dependencies, not simply around available assessment functions.

For an AI provider, the immediate design question is how engineering and compliance teams agree on system qualification before committing to testing. A shared system description can help connect product characteristics to assessment choices, provided that classification and applicable obligations remain subject to appropriate review.

For a compliance technology vendor, the architecture suggests linking system cards, selected controls, technical evidence, stakeholder interpretation, and reporting. An extensible test catalog remains valuable, but the surrounding workflow determines whether its results can be interpreted in the relevant regulatory context.

For a regulatory authority, the six-phase framework provides a candidate structure for specifying information handoffs and supervisory touchpoints. It does not imply that every authority should adopt an identical technical implementation.

A reasonable evaluation criterion for such platforms is whether each participant can establish why an assessment was selected, inspect its evidence, and understand how that evidence informs the next decision.

These are design implications derived from the proposed architecture. The paper does not verify that implementing them reduces regulatory risk or produces a reliable audit trail in production.

Prototype feedback is not operational validation

The study’s evidence supports requirements discovery and the feasibility of developing a more comprehensive collaborative assessment prototype. It does not establish the effectiveness of that prototype under real sandbox operating conditions.

Three boundaries are particularly relevant.

First, the researchers did not conduct sustained deployment within an operating national AIRS. Demonstrations and workshops cannot establish how the system performs under prolonged supervisory work, organizational constraints, or repeated assessment cycles.

Second, the evaluation provides no standardized usability measurements, quantitative effectiveness outcomes, or controlled comparison against existing assessment workflows. Confidentiality restrictions also prevent publication of raw stakeholder quotations and detailed organizational cases, limiting independent verification of the qualitative interpretations.

Third, interoperability remains a proposal. Consistent representations might support collaboration across national sandboxes, but differences in legal practice, institutional capacity, and supervisory procedures could constrain portability.

A stronger evaluation would need to examine actual assessment cases, measure process and decision outcomes, and test whether shared representations maintain their usefulness across different authorities and participating organizations.

Regulatory collaboration begins before test execution

The study identifies a design requirement that conventional testing platforms can overlook: assessment evidence becomes useful for regulatory collaboration only when stakeholders can establish what it addresses and how it should be interpreted.

Its contribution is a concrete, stakeholder-informed architecture connecting qualification, regulatory controls, technical execution, collaborative review, and reporting across a proposed sandbox lifecycle.

For organizations building or procuring AI compliance technology, the relevant question is whether the platform supports those connected decisions and the people accountable for them.

The proposed artifact provides a plausible implementation direction. Demonstrating that it improves regulatory assessment will require evidence from operating sandboxes.

Cognaptus: Automate the Present, Incubate the Future.