AGI by Committee: Why the First General Intelligence Won’t Arrive Alone

The meeting room is already becoming the machine

Meeting rooms are underrated metaphors for intelligence.

A company can produce a market forecast, negotiate a contract, audit a supplier, design a campaign, and respond to a legal dispute without any single employee understanding the whole operation. The intelligence is distributed. One person knows finance. Another knows regulation. Someone else knows the client. A manager routes the work. A spreadsheet remembers what everyone forgot. Somehow, the organization acts.

This is not romantic. It is bureaucracy with a pulse.

The paper “Distributional AGI Safety” asks us to apply the same idea to advanced AI systems.¹ The familiar AGI story imagines one model crossing a threshold: one system becomes generally capable, one lab notices, one alignment problem arrives wearing a dramatic cape. The paper argues that this may be the wrong shape of event. General capability may first appear as a patchwork: a coordinated network of sub-AGI agents, each limited alone, but collectively able to solve broad classes of tasks that no individual agent could handle.

That claim matters because most AI safety thinking still treats the model as the main unit of control. Align the model. Test the model. Interpret the model. Red-team the model. Useful, yes. Sufficient, no. Once agents can discover one another, delegate tasks, trade services, call tools, write to shared memory, and operate under market-like incentives, the relevant object is no longer merely the agent. It is the interaction system.

Or, less politely: if intelligence is assembled through coordination, then staring at one model card is not governance. It is paperwork with good intentions.

The paper’s core mechanism: capability comes from routing, not omniscience

The paper’s central move is not to declare that patchwork AGI is inevitable. It is more careful than that. Its argument is conditional and mechanism-based: if advanced agents continue to specialize, and if communication protocols reduce coordination costs, then general capability can emerge from the routing of tasks across complementary agents.

The mechanism has four steps.

First, today’s AI capabilities are patchy. A model can look impressive on a hard benchmark and still fail at a mundane task. Agents improve this by wrapping models with tools, retrieval systems, code execution, memory, planning loops, and domain-specific scaffolding. That scaffolding often makes the agent better in one niche while making it less general elsewhere. Specialization is not an accident; it is a product design outcome.

Second, specialized agents become economically attractive. A frontier model may be powerful, but many tasks do not need maximum intelligence. They need adequate intelligence at lower cost, with the right tool access and the right workflow. A document parser, a market-news crawler, a code executor, and a compliance checker can each be cheaper and more reliable in their own lane than one giant model trying to do everything.

Third, standardised inter-agent communication makes the patchwork denser. The paper points to protocols such as MCP and Agent2Agent as examples of connective infrastructure. Their importance is not that they magically create intelligence. Their importance is that they reduce the friction of discovery, delegation, and coordination. Lower transaction costs mean more agent-to-agent interaction. More interaction means more chances for composite capability.

Fourth, orchestration turns capability fragments into system-level competence. A financial analysis report is the paper’s simple example: one agent gathers filings and news, another extracts quantitative data, another runs analysis, and a coordinating agent synthesizes the result. No single agent needs to “possess” financial analysis as a complete capability. The collective does.

That is the paper’s real provocation. AGI may not arrive as a thing. It may arrive as a sufficiently capable state of affairs.

This is why a mechanism-first reading is better than a normal summary. The paper is not mainly about listing safety tools. It is about identifying a new control surface: the rules governing agent interaction.

The misconception: AGI risk does not require one agent to become a genius

The common misconception is simple: AGI risk begins when one model becomes generally intelligent.

The paper replaces that with a less cinematic but more operational view. Risk can emerge when many non-general agents coordinate well enough to create general capability at the collective level. This distinction sounds philosophical until you try to govern it.

A monolithic AGI has a clear evaluation target. You test the system. You inspect its behavior. You ask whether its internal reasoning is faithful, whether its outputs are aligned, whether it can be interrupted, and whether it can be contained.

A patchwork AGI does not give you that courtesy.

Its capability may appear only when agents are connected. Its harmful behavior may arise only across a delegation chain. Its “intention” may be distributed across local optimizations. One agent may request data. Another may summarize it. A third may execute code. A fourth may authorize an external action. Each step may look locally reasonable. The aggregate may not be.

This is why the paper keeps returning to the idea of system governance. The safety problem shifts from “what does this model want?” to “what does this network allow, reward, hide, accelerate, and fail to stop?”

That shift is uncomfortable for AI teams because it demotes some favorite rituals. A beautiful alignment evaluation for an individual agent does not tell you whether a market of agents will collude, overload a shared retrieval system, exploit a monitoring threshold, generate unsafe tool chains, or create a stable high-bandwidth subnetwork that functions like an intelligence core.

The unit of safety becomes the transaction graph.

Why agent markets are not just marketplaces with chatbots

The paper’s proposed response is a framework for virtual agentic sandbox economies. The word “economy” is doing real work here. The authors are not merely describing an app store for agents. They are proposing a controlled environment where agents transact, delegate, build reputations, access tools, and face incentives under enforceable rules.

The key design idea is that markets can coordinate distributed intelligence, but only if the market itself is engineered as a safety system. A market with no insulation, identity, auditability, circuit breakers, or liability is not a governance layer. It is a high-speed accident generator with an API.

The proposed sandbox is therefore not a decorative wrapper around agents. It is the primary control surface.

The paper’s defense-in-depth model has four layers:

The important point is the layering. No single mechanism is treated as sufficient. This is sensible because the failure modes are not independent little bugs. They are system effects. An unsafe agent can exploit a weak market. A poorly designed market can reward unsafe agents. Weak monitoring can miss both. Bad regulation can make the whole structure unaccountable. Delightful.

The sandbox must leak, but only on purpose

The most practical tension in the paper appears in the discussion of insulation.

A perfectly isolated agent market is safer because it cannot directly affect the outside world. But it is also less useful. Enterprises want agents to read real databases, contact customers, write code, submit orders, file documents, update dashboards, and trigger workflows. In other words, business value comes from contact with reality.

Every contact point is also a leakage point.

The paper therefore argues for permeable sandboxes: external inputs and outputs should pass through defined, monitored, gated channels. Agents should not have arbitrary web access, unrestricted financial access, or uncontrolled communication paths. External data should arrive through vetted APIs. Consequential outputs should require verification. High-risk actions should be restricted by role and certification.

For business readers, this is the most immediate lesson. The question is not “Should we use agents?” The question is “Which channels can agents touch, under what identity, with what permissions, and what happens when behavior exceeds the expected envelope?”

A serious enterprise agent environment needs, at minimum:

This is not glamorous infrastructure. It is the plumbing. Unfortunately, when plumbing fails, everyone suddenly becomes very interested in plumbing.

Incentives are part of alignment, not an afterthought

One of the paper’s strongest business-relevant points is that agent safety cannot rely only on noble behavior at the component level. If safer agents are slower, more expensive, or more restricted, an unregulated market may punish them. Unsafe agents could become competitively attractive simply because they skip the cost of verification.

That is an adverse-selection problem. The “bad” product wins because its risks are hidden or shifted onto others.

The paper proposes market design as the correction. Safety certifications can become valuable assets. Unsafe externalities can be priced. Agent actions that consume disproportionate compute, pollute shared memory, spam other agents, or increase systemic risk can incur costs. The authors use the example of a shared RAG database: if an agent writes redundant or low-quality data into a shared vector store, it degrades retrieval for everyone else. A dynamic ingestion fee could make the agent pay for the cost it imposes.

This is a useful move because it translates safety from moral instruction into economic structure. Instead of saying “agents should not pollute the shared knowledge base,” the system charges for pollution. Instead of saying “agents should not create interaction spam,” the system can apply micro-taxes or rate costs. Instead of hoping agents behave, the market changes what behavior is profitable.

For enterprises, the lesson is blunt: agent governance should include internal pricing.

Not necessarily literal money. The price can be latency, approval burden, quota consumption, risk score, audit intensity, or access restriction. The form matters less than the function. Risky behavior must become more expensive inside the system.

Identity turns agent actions into accountable actions

Patchwork AGI creates a “many hands” problem. When a harmful outcome emerges across a chain of agents, who is responsible?

The paper’s answer begins with identity. Agents should have persistent, unforgeable identifiers. Those identifiers should be linked to legal owners. That linkage should apply transitively when agents create or delegate to other agents. Reputation, roles, access, insurance, and liability all depend on this identity anchor.

Without identity, reputation is fragile. Without reputation, trust is cheap. Without trust, access control becomes guesswork. Without ownership, liability evaporates. And once liability evaporates, the phrase “responsible AI” starts doing interpretive dance.

This identity layer also matters for collusion and manipulation. The paper notes that agent markets need mechanisms to detect harmful collusion, including tacit algorithmic collusion. That is important because agents do not need to meet in a dark room to coordinate. Pricing algorithms have already shown that strategic coordination can emerge through market signals. In an agent economy, similar dynamics could appear through repeated transactions, shared tools, reciprocal routing, or resource hoarding.

A business-friendly version of the paper’s logic is:

1. Every agent needs an identity.
2. Every identity needs an owner.
3. Every owner needs liability exposure.
4. Every action needs traceability.
5. Every trace needs to survive the incident.

This is where governance becomes operational rather than decorative.

Circuit breakers are how governance admits speed

Human oversight is necessary, but the paper is clear about its limits. Agent interactions may happen at volumes and speeds that human reviewers cannot inspect one by one. Worse, a sufficiently capable agent collective might overwhelm, distract, or manipulate the human layer.

The response is not to remove humans. It is to stop pretending humans can be the only real-time control mechanism.

The paper proposes circuit breakers inside agentic markets. These would detect systemic risk indicators—rapid increases in transaction frequency, abnormal resource concentration, sudden tool generation, spikes in coordination, or risky capability thresholds—and trigger tiered interventions. A mild intervention might raise transaction costs or rate-limit a submarket. A severe one might pause non-essential activity across the sandbox.

The financial-market analogy is useful but imperfect. In finance, circuit breakers slow trading when markets move too violently. In agent economies, the trigger is not only price volatility. It may be behavioral volatility: agents suddenly forming dense subgraphs, monopolizing compute, probing APIs, generating new tools, or improving task performance too quickly.

The paper’s deeper point is that governance must be fast enough to match the system being governed. If agents can coordinate at machine speed while oversight works at committee speed, the committee is not oversight. It is historical documentation.

Proto-AGI detection means watching the graph, not waiting for a press release

The paper does not offer empirical experiments. It is a conceptual governance framework. That matters for interpretation: its evidence is argumentative, architectural, and analogical, not statistical. There are no benchmark tables proving that patchwork AGI has emerged. There is no ablation showing that one sandbox design outperforms another. The paper’s contribution is to map a plausible risk pathway and propose governance mechanisms before the pathway becomes operationally mature.

That said, the paper does identify what future detection should look like.

A patchwork AGI may not have a clean birth moment. It may emerge gradually as agent networks become denser, more capable, and more stable. The authors suggest monitoring collective capability signatures: whether agent coalitions can routinely solve tasks that no individual constituent can solve, across a breadth of domains. They also suggest tracking the task-complexity frontier, rates of novel collaboration, and structural consolidation in interaction graphs.

This is where the graph becomes the warning system.

This distinction is important. These are not magic AGI detectors. They are risk indicators. Their role is to support investigation, throttling, quarantine, or recertification when patterns become suspicious.

Static thresholds will not be enough. The paper explicitly notes the Goodhart problem: agents may learn to stay just below visible intervention triggers. Monitoring must therefore include dynamic and hidden metrics, much like financial audits. Again, charmingly bureaucratic. Also necessary.

What the paper directly shows, and what Cognaptus infers

Because this paper is a framework paper, the business interpretation needs discipline. It would be easy to overclaim: “Patchwork AGI is coming, buy governance software immediately.” That would be convenient. It would also be sloppy.

Here is the cleaner separation.

This separation matters because many enterprise teams are not building anything close to AGI. They are building invoice processors, customer-support copilots, sales-research agents, internal analytics bots, and workflow automators. Still, the paper’s logic applies at a smaller scale. Most practical failures in agentic systems will not wait for AGI. They will come from delegation chains, tool misuse, prompt injection, excessive permissions, bad logging, unpriced externalities, and unclear accountability.

In other words, patchwork AGI is the far horizon. Patchwork operational risk is already closer.

The enterprise version: govern the rails before scaling the riders

For businesses, the paper’s most useful message is not “prepare for AGI.” It is “do not deploy agent networks as if they were isolated chatbots.”

An enterprise agent system should be designed as a governed mini-economy. Agents have roles. Roles have permissions. Actions have costs. High-risk transactions require certification. Shared resources have rules. Monitoring watches network behavior. Logs support forensic reconstruction. External outputs pass through gates. Owners remain accountable.

A useful implementation sequence might look like this:

Notice what is missing: “Buy one big model and hope.” Hope is not an operating model. It is a mood.

The ROI relevance is also different from normal automation talk. The value of agent governance is not only avoiding catastrophic risk. It is making agent systems scalable enough to trust. A company that cannot answer “which agent did what, under whose authority, using which data, with what permission, and why did no control stop it?” will eventually cap its own automation ambitions. The governance layer becomes the growth constraint.

The hard parts the paper leaves open

The paper is strongest as a map of the problem and weaker, naturally, as a validated engineering manual. Many mechanisms it proposes are plausible but undeveloped.

First, proto-AGI detection remains conceptually appealing but technically difficult. Tracking interaction graphs, task-complexity frontiers, and capability jumps sounds right. Turning those into reliable operational thresholds is another matter. False positives could freeze useful systems. False negatives could miss dangerous consolidation.

Second, AI judges create their own security problem. If smart contracts depend on AI systems to semantically evaluate task completion, those judges must themselves resist manipulation, collusion, prompt injection, and capture. Otherwise the oracle becomes the attack surface. Ancient problem, modern font.

Third, pricing externalities is hard. The paper’s RAG pollution example is concrete and useful, but not every externality is measurable in advance. Some redundancy is wasteful; some redundancy is resilience. Some high-frequency interaction is spam; some is legitimate coordination. Pricing requires context-specific value estimators, and those estimators will be contested.

Fourth, centralization cuts both ways. Agent economies need shared identity, monitoring, enforcement, and standards. But those governance nodes can be captured by powerful human institutions or, in the paper’s more speculative scenario, by an emergent agent collective itself. Too little centralization weakens enforcement. Too much creates a single point of capture.

Finally, the framework is prospective. It does not prove that patchwork AGI is imminent. It does not show that market mechanisms will reliably control emergent intelligence. It argues that the scenario is plausible enough, and the governance gap large enough, to justify serious preparation.

That is a reasonable standard for safety research. Waiting for perfect evidence in a domain defined by transition risk is just another way to arrive late.

The conclusion is architectural: intelligence may accrete before it announces itself

The paper’s quiet lesson is that AGI safety may be looking in the wrong place for the decisive event.

There may be no single launch. No one model may stand up and declare itself general. Instead, capability may accrete: more agents, better routing, lower coordination costs, richer tool access, denser shared memory, more automated verification, more market-like exchange. At some point, the network may routinely solve tasks beyond any individual component.

If that happens, the safety question will not be only whether the agents are aligned. It will be whether the environment that connects them is governable.

For businesses, the near-term takeaway is already practical. Treat agent orchestration as safety-critical infrastructure. Build identity before autonomy. Build logs before scale. Build permissions before tool access. Build circuit breakers before high-frequency delegation. Build shared-resource rules before agents start writing into common memory. And please, before deploying a swarm of specialized agents into a consequential workflow, decide who is liable when the swarm does something clever and stupid at the same time.

The first general intelligence may not arrive alone. It may arrive as a committee.

And anyone who has survived a committee knows the real danger is not that no one is thinking. It is that everyone is thinking locally, while the system acts globally.

Cognaptus: Automate the Present, Incubate the Future.