Concurrency, But Make It Fashion: Why Trustworthy AI Needs an Agentic Lakehouse

Every enterprise AI conversation eventually reaches the same awkward sentence: “Yes, the agent can write code, but absolutely do not let it touch production.”

This is not because executives have suddenly become philosophers of machine autonomy. It is because production data is where optimism goes to be audited. A clever agent that drafts SQL, patches a pipeline, or debugs a transformation is useful right up to the moment it drops a table, joins incompatible versions of data, installs a charmingly malicious package, or writes hallucinated output into a dataset used by finance, compliance, or customer operations. At that point, it is no longer “agentic productivity”. It is an incident report with better syntax.

The paper Trustworthy AI in the Agentic Lakehouse: from Concurrency to Governance makes a useful, unfashionably infrastructural argument: trustworthy enterprise agents will not come primarily from better prompting, friendlier approval buttons, or asking the model to “be careful”. They require a lakehouse architecture where agent work is isolated by construction: data snapshots, Git-like branches, atomic merge, network-isolated function execution, declarative I/O, and a narrow run API that can actually be governed.¹

That is the paper’s real contribution. It does not say, “Here is another agent that can fix data pipelines.” It says, more interestingly, “Here is why agents should never be allowed to work directly on the lakehouse in the first place.”

A small distinction. Only the usual difference between governance and theatre.

The trust problem begins with side effects, not model intelligence

The default enterprise instinct is to treat AI trust as a model behaviour problem. Was the model accurate? Did it cite sources? Did it follow instructions? Did it apologise in a calm voice after doing something expensive?

Those questions matter, but they are downstream. In data engineering, the dangerous part is not that the agent thinks incorrectly. Humans do that daily and have built entire meeting cultures around it. The dangerous part is that the agent can act incorrectly against shared, persistent infrastructure.

A data pipeline is not a chat response. It has side effects. It reads tables, transforms data, writes intermediate outputs, refreshes assets, triggers dashboards, updates downstream systems, and occasionally poisons an entire analytical workflow while looking quite tidy in the logs. The paper’s central move is to shift the trust discussion from agent psychology to system boundaries.

The question becomes:

Can an untrusted or partially trusted agent work on production-adjacent data without being able to corrupt production state?

That is a much better question than “Can we make the model safer?” because it has an architectural answer. The paper argues that the answer starts with concurrency control: the old database problem of allowing multiple users to read and write safely at the same time.

In other words, trustworthy agents need the thing databases already learned the hard way: the illusion of freedom, surrounded by a very serious cage.

MVCC is a mental model, not a lakehouse copy-paste job

The authors use multi-version concurrency control, or MVCC, as the motivating analogy. In a transactional database, each user can behave as if they are operating alone, while the system manages concurrent reads and writes underneath. Readers see a consistent snapshot. Writers either publish a complete transaction or publish nothing. The user does not manage the internal storage mechanics; the database does.

That last point is the important one. MVCC is not just clever storage machinery. It is a governance simplifier because it hides physical complexity behind a logical interface. Users express intent through SQL. The system handles execution, isolation, atomicity, and permissions. RBAC becomes meaningful because access is mediated through a controlled layer. Unauthorised code does not wander directly through physical files with a clipboard and a sense of destiny.

The paper divides the useful MVCC lesson into three parts:

This is where the paper avoids a common trap. It does not pretend that a lakehouse is just a bigger database with worse branding. A transactional database is vertically integrated: storage, compute, and API are tightly controlled. A lakehouse is deliberately decoupled: object storage, table formats, Python functions, SQL engines, notebooks, Airflow DAGs, package dependencies, and orchestration layers all coexist in a somewhat civilised cloud traffic jam.

MVCC is therefore not a solution to transplant. It is a mental model to reimplement.

The paper’s mechanism-first logic is straightforward: if the lakehouse can recreate the operational properties that made MVCC governable, then agent governance becomes tractable. If it cannot, then “AI governance” becomes a layer of policy documents floating above systems that still allow unsafe side effects.

A familiar enterprise arrangement, sadly.

The lakehouse breaks isolation in three different ways

The paper’s diagnosis of the standard lakehouse is useful because it does not rely on vague “complexity”. It names the failure modes.

First, table-level transactions are not enough. Many lakehouse storage systems provide transaction-like guarantees at the table level. That sounds comforting until the workload is a pipeline. A pipeline writes across multiple nodes and tables. If an agent writes faulty code halfway through, earlier nodes may commit successfully while later nodes fail. Each individual table operation can be technically valid, while the lake as a whole becomes inconsistent.

That is the sort of correctness failure that looks fine in a narrow transaction log and wrong in the business dashboard. Excellent news for anyone who enjoys root-cause analysis after lunch.

Second, compute is heterogeneous. One node might run SQL, another Python 3.10 with pandas, another Python 3.11 with polars. This heterogeneity is normal in modern data engineering. It is also awkward for governance because every runtime, dependency manager, package source, execution environment, and network access path becomes part of the risk surface.

Third, programming abstractions are scattered. In a conventional pipeline, the user often controls both the logic and the physical side effects: read from this S3 path, write to that bucket, return “SUCCESS”, and hope the surrounding orchestration interprets reality generously. The paper uses an Airflow-style example to make the point: if a function writes output as a side effect and fails just before returning, the platform has no principled transactional boundary around the pipeline.

This is the core failure. The lakehouse does not merely lack one feature called “agent safety”. It lacks a unified isolation contract across data, compute, and API access.

That is why human approval at the end is not sufficient. Approval is only useful if the thing being approved is already contained, inspectable, and reversible. Otherwise the human is not reviewing a proposed change. They are standing near a live production system hoping the agent has manners.

Bauplan’s design rebuilds trust as infrastructure

The paper proposes Bauplan as a concrete agent-first lakehouse design. The important point is not the product name. The important point is the control pattern.

Bauplan reinterprets MVCC-like guarantees using lakehouse-native primitives:

The mechanism matters more than the vocabulary. Each primitive removes a class of dangerous freedom.

Snapshots protect reads. Branches contain writes. Isolated functions constrain execution. Declarative I/O narrows permissions. Atomic merge turns publication into a governed event. The run API binds the whole sequence so the platform, not the agent, owns the operational contract.

This is the difference between giving an agent shell access to your lakehouse and giving it a controlled workspace where every action has a boundary.

The former is “move fast and apologise to compliance”. The latter has a future.

Branches turn agent work into a reviewable data artifact

The paper’s most business-relevant idea is that agent output should look less like a direct production action and more like a pull request.

In Bauplan’s model, every write is immutably recorded. Snapshots form a history. Branches represent alternative states. Merge resolves publication. This is familiar in software engineering, but the paper applies it to data pipelines. The agent does not “fix production”. It produces a branch containing proposed data and pipeline changes.

That change in object type matters. A branch can be inspected. It can be tested. It can be rejected. It can be merged atomically. It gives human review something concrete to review.

The paper highlights why this is necessary for multi-table pipelines. A single table commit is not the same as a correct pipeline run. Suppose a two-node pipeline produces table A and table B. Without branch-level pipeline isolation, a failed run can leave production with a new version of A and an old version of B. Everything is individually explainable. The whole is wrong.

Bauplan’s run model instead opens a temporary branch, executes the pipeline there, and merges changes into main only on success. On failure, the temporary branch remains available and production remains untouched. That is the transaction boundary the lakehouse was missing: not one table, not one file write, but the whole pipeline run.

The business translation is direct. If agents are going to write or repair pipelines, their work must become a staged operational artifact. Not a wish. Not a Slack message. Not a heroic notebook. A branch.

Sandboxed functions make compute governable

Data isolation alone is not enough because agents do not only choose what to write. They generate code.

The paper uses a Function-as-a-Service model to handle compute isolation. Each pipeline node runs in its own containerized Python or SQL environment, independent of other functions and isolated from the public internet. Agents provide the business logic; the platform controls hardware, runtime, operating system, package environment, and execution boundaries.

This division of labour is the governance move. It prevents the agent from treating the lakehouse as a general-purpose computing playground. That matters because Python is both useful and delightfully irresponsible when left unsupervised. Package installation, network calls, environment drift, hidden side effects, and runtime inconsistencies are not edge cases. They are ordinary sources of data engineering fragility.

In the proposed design, packages are still a major risk vector, but they become governable because dependencies are declared. A platform can whitelist allowed packages, reject unsafe decorators, and prevent arbitrary internet access. This is not glamorous, but neither is incident prevention. That is rather the point.

The paper’s contribution here is not that containers exist. The contribution is placing containerized execution inside a larger transactional pattern: branch, run, verify, merge. Compute isolation is useful because it is tied back to data isolation and API governance.

A sandbox without a publication boundary is just a cleaner place to make a mess.

Declarative I/O narrows the blast radius

The paper’s treatment of declarative I/O is easy to underestimate. It is not just syntactic neatness. It is where governance becomes enforceable.

In conventional pipeline code, a function may read from physical paths and write to buckets or prefixes directly. That gives the agent access to the physical layer. It also forces governance to chase every possible side effect: files, paths, engines, credentials, packages, orchestration, and exploratory query access.

Bauplan’s alternative is to make functions accept tables as inputs and return tables as outputs. The code declares its runtime and dependencies through decorators. Business logic sits inside a function, but I/O is platform-mediated.

That changes the permission model. Instead of asking, “What can this arbitrary script do once it has credentials?”, the platform can ask narrower questions:

This is why the paper keeps returning to API surface area. A broad surface is not merely harder to document. It is harder to secure. Every extra interface is another place where the agent’s operational freedom can escape the policy model.

The argument is not anti-flexibility. It is anti-accidental flexibility. Enterprises do not need agents that can do anything. They need agents that can do the right class of things inside boundaries that survive bad code, bad reasoning, and bad Tuesday mornings.

The run API is where governance stops being decorative

The paper’s unified run API is the binding mechanism. During a run on the main branch, Bauplan automatically opens a temporary branch, fetches the relevant source rows, executes user code in isolated FaaS runtimes, writes outputs to storage, and either merges the temporary branch into main on success or leaves it separate on failure.

That sequence matters because governance is no longer an external approval ritual. It is built into the lifecycle of execution.

The agent cannot simply write into production because the run begins on a branch. It cannot quietly install arbitrary packages because the runtime is declared. It cannot freely access the physical storage layer because I/O is mediated through tables. It cannot partially publish a failed pipeline because merge is atomic. It cannot bypass all of this without leaving the governed API surface, which is exactly where RBAC and platform controls should apply.

This is the paper’s strongest causal chain:

1. Concurrency requires isolation.
2. Isolation requires controlled data and compute boundaries.
3. Controlled boundaries require declarative abstractions.
4. Declarative abstractions create a narrow API surface.
5. A narrow API surface makes governance enforceable.

That is more useful than saying “AI needs governance”, which is true in the same way that buildings need gravity. Yes. We noticed.

The paper’s deeper point is that governance becomes tractable only when the system structure reduces the number of things governance must understand.

The self-healing pipeline is implementation evidence, not a benchmark

The worked example in the paper is a self-repairing pipeline. A pipeline run fails. A human engineer prompts an agent for a resolution and provides a verifier, meaning computational acceptance criteria registered in the platform. The agent uses a ReAct-style loop over Bauplan APIs to inspect, reason, and propose a fix. Its code runs in the sandbox. Its writes happen on a branch. The verifier checks minimal criteria. A human reviews and confirms before merge.

This example is valuable, but it should be interpreted correctly.

It is not a benchmark showing that Bauplan agents outperform other agents across many pipeline failures. It is not a robustness study across vendors, workloads, adversarial prompts, or messy enterprise schemas. It is a reference implementation demonstrating how the architectural primitives combine into a safe operational flow.

That still matters. In fast-moving AI infrastructure, a worked implementation can be more informative than another diagram promising “end-to-end governance” with arrows of destiny. The self-healing example shows the control sequence:

The paper is therefore best read as an architecture argument with a working pattern, not as a statistical evaluation paper. That boundary is not a weakness if understood properly. It means the practical question is not “What is the reported accuracy?” but “Does this design remove the right failure modes from the agent’s direct control?”

For many enterprise data teams, that is the more urgent question anyway.

Business value: smaller governance surfaces, safer autonomy

The business relevance of this paper is not that every enterprise should immediately adopt Bauplan. The stronger interpretation is that any serious agentic data platform should expose similar control primitives.

The practical pathway looks like this:

1. Do not give agents direct production write access.
2. Route all agent work through temporary branches or equivalent isolated states.
3. Make pipeline publication atomic across all affected tables.
4. Run generated code in sandboxed environments without broad internet access.
5. Force data access through declarative table-level I/O.
6. Use verifier checks before human review.
7. Treat merge as the operational permission boundary.

This changes the economics of agent adoption. Without such infrastructure, every new agent capability expands the risk surface: more tools, more credentials, more runtime variation, more review burden. With isolation and declarative interfaces, agent autonomy can increase without requiring governance teams to inspect every internal move manually.

That is the ROI argument. Not “agents replace data engineers”. That line is catnip for demos and indigestion for operations. The better argument is that agents can handle more debugging, repair, and iteration when their work is staged, testable, and reversible.

The value is not full autonomy. It is safe parallelism.

If one human engineer can supervise several agent-generated repair branches, each guarded by sandboxing, declarative I/O, verifier checks, and atomic merge, then the organisation gets leverage without pretending that the agent has become a senior platform engineer with impeccable judgement and a pension plan.

What this means for enterprise architecture decisions

The paper gives buyers and builders a useful evaluation lens. When a vendor claims to offer “agentic data engineering”, the question should not begin with the model name. It should begin with the isolation contract.

A useful procurement checklist would ask:

This is also a useful internal architecture test. If a company’s agentic data workflow still relies on notebooks with broad credentials, orchestration scripts with side effects, uncontrolled package access, and manual review after production mutation, then the issue is not that the model is insufficiently aligned. The issue is that the organisation has built a haunted house and invited the robot in first.

The main uncertainty is empirical, not conceptual

The paper is a position paper with a concrete system design and reference implementation. It does not provide a broad empirical evaluation showing reliability gains across many enterprises, clouds, data formats, orchestration systems, schemas, agent models, or adversarial conditions. That limits what we can claim.

What the paper directly shows is an architectural mapping: MVCC-inspired ideas can be reimplemented for an agentic lakehouse through branches, isolated FaaS, declarative I/O, and a unified run API. It also shows a worked self-healing pipeline pattern demonstrating how agent reasoning, verifier checks, human review, and atomic merge can fit together.

What Cognaptus infers for business practice is that agentic data engineering should be governed at the infrastructure layer before it is governed at the policy layer. If the platform cannot isolate reads, contain writes, sandbox execution, and mediate I/O, then governance has to chase behaviour after the fact. That is expensive, brittle, and usually late.

What remains uncertain is operational breadth. How well does this pattern scale across messy legacy data estates? How easy is verifier authoring for non-trivial semantic correctness? How do conflict resolution, cost control, lineage, privacy constraints, and cross-platform orchestration behave under heavy agent concurrency? How much developer experience friction appears when teams move from flexible scripts to stricter declarative APIs?

Those are adoption questions, not reasons to ignore the design. The paper’s point is that without this class of isolation machinery, the alternative is worse: trusting agents directly with production data and calling the resulting anxiety “human oversight”.

The real lesson: trust is a transaction boundary

The fashionable version of agentic AI imagines more capable models roaming through enterprise systems, interpreting goals, fixing problems, and taking action. The unfashionable version asks where their writes go, what runtime they use, what packages they install, whether production is touched before review, and how failure is contained.

The unfashionable version is the one that will survive procurement.

This paper’s durable insight is that trustworthy AI in the lakehouse is not mainly a question of model confidence. It is a question of transaction boundaries. Agents need places to think, test, fail, and repair without corrupting shared state. They need branches, sandboxes, declarative interfaces, verifier hooks, and merge gates. They need infrastructure that assumes partial untrustworthiness and still permits useful work.

That is not a pessimistic view of AI. It is a professional one.

The future of enterprise agents will not be decided only by who has the smartest model. It will be decided by who gives those models the safest operational surface. Concurrency first. Governance follows. Fashion, for once, can wait.

Cognaptus: Automate the Present, Incubate the Future.