Curiosity Under Constraint: Engineering Agency, Not Just Intelligence

A good assistant is not always the one that answers fastest.

Sometimes it should ask for another file. Sometimes it should stop reading and act. Sometimes it should think privately for a few more steps. Sometimes it should say nothing, because another paragraph of “reasoning” would merely burn tokens while impressing nobody except the invoice.

This is the useful irritation behind Richard Csaky’s working draft, Artificial Agency Program: Curiosity, compression, and communication in agents.¹ The paper is not another sermon about making models larger, more autonomous, or more verbally thoughtful. The industry has already tried “more” as a strategy. It is easy to understand, easy to fund, and wonderfully convenient for slide decks.

The paper asks a sharper question: what should an agent spend its limited resources on?

That question changes the unit of analysis. The system is no longer just a model producing outputs. It is an embedded agent operating inside a human–tool–environment loop, with limited sensing, limited action authority, finite memory, compute costs, communication costs, and incomplete knowledge of the world. In that setting, intelligence is not merely the ability to solve a task. Agency is the ability to allocate scarce observation, action, deliberation, and communication toward useful prediction and control.

That is the mechanism worth following.

The core claim is cost-sensitive agency, not bigger autonomy

The most common misreading of agent papers is predictable: more autonomy equals more progress. Give the system more tools, longer context, more private reasoning, more browser access, more memory, and perhaps a small digital badge saying “agentic.” Then wait for productivity.

AAP moves in the opposite direction. Its claim is not that agents should always see more, act more, think more, or speak more. Its claim is that each of these activities has a cost, and an agent should spend that cost only when the expected gain in learning progress, prediction, control, or coordination justifies it.

The paper’s definition of curiosity is central here. Curiosity is not raw novelty. A random stream of noise is novel, but it teaches nothing. A fully mastered pattern is predictable, but no longer useful for learning. AAP adopts curiosity-as-learning-progress: the agent is intrinsically rewarded when its predictive model improves, especially over future observations.

In plain terms, the agent should be drawn toward situations where it can still learn something meaningful. Not the impossible. Not the trivial. The learnable frontier.

For business systems, that distinction matters. A market-monitoring agent that chases every strange price movement is not curious; it is distracted. A customer-support agent that repeatedly asks for information it already has is not careful; it is expensive. A research agent that reads ten more papers after the decision-relevant uncertainty has already collapsed is not rigorous; it is procrastinating with citations.

AAP’s stronger move is to combine learning progress with explicit costs. Observation has a cost. Action has a cost. Compute and deliberation have a cost. Memory maintenance has a cost. The paper’s formal objective subtracts these costs from the learning-progress reward. The exact notation matters less than the accounting discipline: the agent’s intelligence is evaluated through budgeted interaction, not isolated answer quality.

Resource constraints create the bottleneck that makes agency measurable

The paper begins from a practical observation: real agents do not operate in frictionless benchmark space. They interact with environments through constrained interfaces.

A robot sees only through its sensors. A trading system sees only through available feeds and latency windows. A document-review agent sees only the documents, metadata, and retrieval channels it can access. A manager using an AI assistant sees only what the interface makes visible. The agent’s “world” is not the world. It is the world filtered through observation channels, action channels, memory, compute, and communication.

AAP treats those limits not as implementation annoyances but as the central design object.

This is where the paper becomes more useful than a conventional “agents are the future” essay. It gives a way to ask whether a system’s poor performance comes from weak reasoning, weak sensing, weak action authority, poor interface design, or bad compute allocation.

That diagnostic separation is important because firms often misdiagnose AI failure. They buy a larger model when the real bottleneck is stale data. They add agent autonomy when the real bottleneck is unclear human approval. They add chain-of-thought-style verbosity when the real bottleneck is a missing API. Very advanced wrong layer. A classic enterprise hobby.

AAP’s mechanism says: first identify which bottleneck prevents learning progress and useful control. Then decide whether relaxing that bottleneck is worth the cost.

Curiosity becomes a pressure toward better interfaces

Once curiosity is defined as learning progress under constraints, the next step follows naturally. If the agent cannot improve prediction because its interface is too lossy, it should prefer interface improvements—if those improvements unlock future learning or control.

The paper calls this “unification”: a task-relative reduction of sensing, acting, and communication bottlenecks between agent and environment. The word is abstract, but the operational idea is simple. A better interface lets the agent’s internal model and actions couple more effectively with the world.

For a physical agent, this might mean better sensors or richer action affordances. For an enterprise AI system, it might mean cleaner data pipelines, lower-latency tool calls, better document retrieval, clearer human feedback, or safer execution permissions. The interface is not decoration around intelligence. It is part of the agent’s effective intelligence.

AAP’s second hypothesis predicts that an agent able to invest in its interfaces should allocate resources toward improvements that increase long-horizon learning progress and control, until costs dominate. This last phrase is doing real work. The paper does not claim infinite integration is good. A perfect data pipeline that costs more than the decision it improves is not “unified.” It is just expensive plumbing with a philosophical accent.

A business-friendly version of the mechanism looks like this:

Resource constraint
        ↓
Bottleneck in observation, action, compute, memory, or communication
        ↓
Reduced learning progress or control
        ↓
Agent has incentive to improve the limiting interface
        ↓
Improvement continues only while marginal gains exceed marginal costs

This is why the article should not be read as a manifesto for fully autonomous systems. It is closer to a budgeted theory of agency architecture. Sometimes the right design is more autonomy. Sometimes it is better retrieval. Sometimes it is less verbal reasoning. Sometimes it is a stricter permission boundary because cheap action can create expensive mistakes.

Prediction and control are related, but not identical

AAP links predictive compression to empowerment and control. Empowerment, in information-theoretic terms, measures how much an agent’s actions can influence future observations. Plasticity, as used in the paper, captures the degree to which observations influence actions, with directed information proposed as a proxy.

The useful insight is bidirectional coupling. A good agent should not merely predict the world like a passive analyst. It should also understand which actions can change future states. But control without prediction is also dangerous. It can become brittle exploitation: the system can push buttons without understanding the situation it is pushing them into.

The paper is careful here. It does not claim prediction, hidden-state compression, empowerment, and task reward are always equivalent. It explicitly allows regimes where prediction improves without useful control, or where control improves through narrow exploitation that does not build broader predictive competence.

That boundary is important for business readers.

A sales automation system may predict lead conversion well but have little ability to change outcomes if the product-market fit is weak. A logistics agent may have strong action authority but poor prediction under supply shocks. A trading bot may control order placement perfectly while misunderstanding regime change, which is a very elegant way to automate loss.

AAP’s first hypothesis is therefore pragmatic rather than absolute: across a substantial regime of tasks and constraints, interventions that improve learning progress should tend to improve useful control, and vice versa. “Tend” is not a decorative hedge. It is the whole point. The paper is proposing a testable alignment between objectives, not a metaphysical identity.

The hypotheses are proposed tests, not reported victories

The paper is a position and research agenda, not an empirical paper reporting a completed benchmark suite. That distinction should not be blurred. The valuable contribution is that it turns the mechanism into falsifiable hypotheses and an experimental roadmap.

This table is where the paper becomes experimentally honest. It includes disconfirmation paths. For example, H4 fails if tuned static schedules match adaptive meta-control after accounting for overhead. H5 fails if private self-communication degenerates into redundant verbosity or loses to latent recurrence under matched compute.

That matters because the AI industry often treats “agentic” features as self-justifying. AAP says: no, the features must earn their budget.

Language is one channel, not the throne of thought

One of the paper’s more useful sections concerns language. AAP treats language as a selective, lossy, resource-constrained communication channel. This is a good antidote to a confused debate around reasoning models.

The paper separates two questions that are often fused together:

1. Is language useful for acquiring broad abstractions during training?
2. Is language the best online medium for every intermediate computation during action?

The answer to the first can be yes while the answer to the second is no. A model may learn a great deal from language, yet still benefit from non-verbal internal deliberation once deployed. Words are useful. Words are not magic. This will disappoint both mystics and prompt engineers, though for different reasons.

AAP proposes that future multimodal systems should be able to interleave observation tokens, action tokens, and private deliberation tokens across modalities. A system might observe visual input, perform silent latent updates, emit an action, generate text for a human, then pause for additional internal computation. The design question is not whether private tokens should exist. It is when they are worth emitting under budget, and which modality should carry the deliberation.

For enterprise AI, this is immediately relevant. Verbose reasoning traces are not free. They increase token costs, latency, storage burden, and audit complexity. But eliminating all intermediate communication can reduce debuggability and coordination. AAP’s answer is not “always show reasoning” or “never show reasoning.” It is: treat communication as an action with expected value and cost.

That is a more mature framing. It also supports a cleaner product question:

The practical lesson is not to worship hidden thought or visible thought. It is to price thought.

The experimental agenda starts with simple worlds for a reason

AAP’s proposed experimental program moves in stages. It begins with synthetic partially observed environments, then moves toward ARC-AGI-style interactive inference, and finally toward multimodal vision-language-action meta-control.

This sequence is sensible. The first stage is not meant to impress anyone with benchmark glamour. It is meant to make variables controllable. In toy POMDPs, researchers can manipulate observation noise, sensor coarsening, latency, action-set size, and cost structure. That makes it possible to test whether the proposed metrics behave as intended.

The second stage, ARC-AGI-style interactive inference, introduces sparse data and compositional generalization. Here, the agent must decide when to request more observation, when to spend compute, and when to act. This is closer to the real economics of problem-solving: information is useful only if it changes the decision enough to justify acquiring it.

The third stage uses a pretrained multimodal model as a backbone, with a lightweight meta-controller deciding whether to observe, act, deliberate, or adapt. This is the bridge to current AI systems. Rather than retraining a giant model from scratch, the paper suggests starting with frozen backbones, lightweight adaptation, recurrent adapters, LoRA-style components, external memory, and explicit meta-control.

The proposed proof-of-concept is especially important because it defines success beyond raw score. The paper suggests comparing adaptive meta-control against fixed observe/act/deliberate schedules; latent-only recurrence against explicit private tokens; and tight versus loose observation/action bottlenecks, with and without deliberation cost penalties.

The primary criteria are frontier improvement at matched cost, interpretable use of private computation, and systematic shifts in meta-action allocation as bottlenecks and costs change. In other words, the agent should not merely perform better. It should spend differently for understandable reasons.

That is the difference between an agent and a very expensive reflex.

Business value comes from diagnosing the limiting budget

The business relevance of AAP is not that companies should immediately build agents using this exact objective. The paper is a research agenda, and several quantities it discusses—empowerment, directed information, interface-quality measures—are difficult to estimate in high-dimensional real systems.

The near-term value is diagnostic.

AAP gives business leaders and system designers a more precise way to ask why an AI workflow fails or becomes uneconomic.

This is a practical shift in procurement and evaluation. Instead of asking only whether Model A beats Model B on a benchmark, firms should ask:

• What does the system need to observe?
• What can it safely change?
• When should it spend more compute?
• What should it remember?
• What should it communicate, to whom, and at what cost?
• Where is the bottleneck between prediction and action?

Those questions sound less glamorous than “agentic transformation.” They are also more likely to survive contact with a budget committee.

Governance through constraints is more concrete than governance through vibes

AAP also has governance implications, though not in the usual abstract sense. If constraints shape behavior, then governance is partly the design of constraints.

A system with unlimited tool access, cheap action, persistent memory, and weak approval boundaries will behave differently from a system where actions are priced, logged, reversible, and permissioned. A system that pays no penalty for asking users unnecessary questions will ask too many. A system that pays no penalty for long private deliberation will deliberate performatively. A system with high action authority and poor observation will confidently do the wrong thing. Progress.

The paper’s constraint-first framing suggests that alignment and governance should not rely only on reward shaping or policy documents. They should be implemented through the economics of the agent’s environment: action costs, observation costs, memory costs, latency costs, communication limits, and escalation rules.

This does not solve safety. It does provide a more engineering-friendly surface. Instead of saying “make the agent responsible,” one can say: restrict irreversible actions, price uncertainty, require higher confidence for high-impact execution, expose observation gaps, and make the agent justify expensive deliberation only when the expected decision value is high.

That is less poetic. Good.

The boundaries are real, and they matter

AAP should be read as a framework, not as evidence that its proposed agents already outperform existing systems. The paper’s own limitations are material.

First, prediction, control, and reward can diverge. Some environments are predictable but uncontrollable. Others are controllable in ways that do not improve broad understanding. Business systems see this constantly: a dashboard can forecast demand without giving the firm the operational ability to meet it.

Second, the metrics are difficult. Empowerment and directed information are elegant in theory, but hard to estimate reliably in messy, high-dimensional enterprise environments. Proxy metrics will be necessary, and proxy metrics bring interpretation risk. The proxy is not the phenomenon. This sentence should be printed above many dashboards.

Third, energy and compute accounting is not simple. Token count, wall-clock latency, FLOPs, and hardware energy are related but not interchangeable. Cross-system comparisons depend on implementation details, deployment scale, caching, parallelism, and pretraining costs.

Fourth, self-communication can become degenerate. Private tokens can help planning, but they can also become verbose internal bureaucracy. Anyone who has watched a model spend 800 words failing to decide whether to call a tool will recognize the problem.

Finally, human-likeness is not automatically safety. AAP’s constraint-manifold view says that systems closer to human constraints may be easier to interpret in some contexts, but similarity is not morality. A human-like failure mode is still a failure mode. Occasionally it is worse, because we find it charming.

The useful shift is from intelligence scores to budgeted agency

The strongest contribution of AAP is not a single formula. It is the shift in evaluation.

Modern AI evaluation still leans heavily toward capability scores: accuracy, benchmark rank, task completion, win rate, leaderboard position. Those metrics are not useless. They are incomplete. They often hide the resource policy that produced the result.

AAP asks us to evaluate the policy underneath the performance:

What did the agent observe?
What did it ignore?
When did it act?
When did it wait?
When did it deliberate?
When did it communicate?
What did each choice cost?
What future prediction or control did each choice buy?

That framing is especially relevant as companies move from chatbots to operational agents. In a chatbot, wasted tokens are annoying. In an operational agent, wasted observation, wrong action, stale memory, and unnecessary latency become process costs. Sometimes legal costs. Occasionally incident reports with very educational subject lines.

A business AI system should therefore be judged not only by whether it can solve a task, but by whether it can solve the task through disciplined resource allocation. The agent must know when better information is worth buying, when action is safe, when private deliberation improves the decision, and when communication helps the human rather than merely decorating the transcript.

That is engineering agency, not just intelligence.

And it is the more serious frontier.

Cognaptus: Automate the Present, Incubate the Future.