Agents Behaving Badly: Why 'Agentic AI' Needs Adult Supervision

A travel agent that books a bad flight is annoying. A travel agent that books the wrong flight, triggers a hotel agent to change the reservation, alerts a finance agent to approve reimbursement, and then lets a calendar agent reschedule meetings around the mistake is no longer annoying. It is an organizational incident with a charming user interface.

This is the uncomfortable point behind Virginia Dignum and Frank Dignum’s paper, “Agentifying Agentic AI.”¹ The paper does not ask whether LLM-based agents are impressive. Of course they are. They can call tools, plan steps, summarize context, and interact in natural language. Very nice. The paper asks a colder question: does this make them agents in the older, stricter sense used by the Autonomous Agents and Multi-Agent Systems, or AAMAS, research community?

Their answer is basically: not yet. What we currently call “agentic AI” often behaves autonomously, but its autonomy is mostly behavioral. It lacks explicit models of cognition, communication, coordination, incentives, norms, trust, and belief. That means it can look agent-like from the outside while remaining difficult to verify, govern, and coordinate on the inside. In business language, it has workflow energy without enough organizational discipline. The intern has keys to the building. Splendid.

The paper is conceptual rather than experimental. There are no benchmark tables showing that one architecture beats another by 7.3 percentage points. There are no ablations, robustness checks, or appendix variants pretending to be a second thesis. Its contribution is different: it takes decades of AAMAS work and uses it as a diagnostic map for what current agentic AI is missing.

That makes the article worth reading not as a “new model” story, but as a systems design warning. If businesses deploy AI agents as if tool use plus memory plus chat equals agency, they will confuse fluency with governability. The paper’s stronger message is that LLMs should not be treated as complete agents. They should be bounded components inside structured agent architectures.

The failure mode is not stupidity; it is unverifiable autonomy

The common criticism of LLM agents is that they hallucinate, forget instructions, or fail at long tasks. Those are real problems, but the paper points to a deeper one: LLMs are not easily treated as predictable, specifiable software modules.

Traditional software components have expected inputs, outputs, and functions. That does not mean they never fail. It means engineers can reason about what the component is supposed to do, what counts as failure, and how the failure should be contained. An LLM is different. It imitates human-like output from patterns in data. It may produce an acceptable result today and a surprising result tomorrow under slightly different context. Prior success does not guarantee future behavior in the way a normal module contract would.

The paper’s travel example is simple but useful. Ask an agent to book conference flights. One time it may choose a morning arrival before the first session and a departure after the final session. Another time it may choose a cheap return flight on the morning of the final day. The second result is “reasonable” in a narrow optimization sense and wrong in the user’s real objective. The issue is not that the model cannot produce text about travel. It is that the goal, constraints, commitments, and stopping conditions are not structurally represented.

In a single-agent setting, the user may catch the error. In a multi-agent system, the bad action can become input for other autonomous actions. The hotel agent adjusts dates. The calendar agent reschedules meetings. The finance agent records expenses. By the time a human notices, the problem is no longer one bad answer; it is a chain of operational side effects.

That is why “agentic” can be worse than passive AI when badly designed. A passive chatbot can be wrong and stay wrong on the screen. An agent can be wrong and then do things.

LLMs should handle practical execution, not own the whole agency stack

The paper’s most useful architectural suggestion is understated: start with traditional agent structures, then use LLMs for specific parts where their flexibility is valuable. In other words, do not make the LLM the sovereign ruler of the workflow. Make it a bounded worker inside a system that has explicit goals, checks, commitments, and permissions.

This is a quiet but important reversal of the usual agentic-AI pitch. The standard pitch starts with the LLM: give it tools, memory, goals, and a loop. The paper starts with the agent architecture: define what the system is trying to achieve, what it believes, what it is allowed to do, when it should stop, how it should communicate, and when humans or other agents must be consulted. Then use the LLM where open-ended language, search, summarization, or messy execution needs flexibility.

For business systems, this difference matters. If the LLM owns the workflow, governance becomes prompt archaeology: when something goes wrong, everyone scrolls through logs trying to determine which instruction failed to hypnotize the model sufficiently. If the LLM is embedded inside a structured workflow, the organization can inspect the explicit parts: goals, roles, permissions, state transitions, validation rules, approval gates, and escalation conditions.

The paper is not arguing for nostalgia. Traditional agent systems were often controllable but brittle because knowledge and intelligence had to be hand-coded. LLMs bring flexibility and broad competence. The point is not to reject them. The point is to stop pretending that flexibility replaces structure.

BDI architecture: give the agent a mind you can inspect

The first AAMAS mechanism the paper highlights is the Belief–Desire–Intention, or BDI, architecture. In a BDI system, beliefs represent what the agent takes to be true about the world; desires represent goals or motivations; intentions represent the subset of goals the agent has committed to pursue.

This sounds philosophical until the flight booking example returns with a clipboard.

A business travel agent should not merely generate plausible booking actions. It should explicitly represent facts, goals, and commitments:

The point is not that every enterprise agent must literally implement a classic BDI system. The point is that some equivalent structure is needed if the organization wants interpretability. Without explicit beliefs, goals, and intentions, the agent’s “intentionality” is inferred from surface behavior. It appears purposeful because the output looks purposeful. That is not the same as having a verifiable mapping from motivation to action.

For business use, this becomes a design rule: every autonomous agent should have an inspectable state model. What does it believe? What goal is it pursuing? What commitment has it made? What condition would make it give up, ask for help, or escalate?

If the only answer is “the prompt says so,” the system is still a chatbot wearing a hard hat.

Communication protocols: natural language is flexible, which is precisely the problem

The second mechanism is communication. Current agentic systems often use natural language as the universal interface between agents. This is attractive because natural language is expressive and cheap to implement. It is also dangerous because it lacks guaranteed semantics.

AAMAS research developed formal communication languages such as KQML and FIPA-ACL, where messages are not just strings but communicative acts: requests, promises, commitments, queries, assertions, refusals, and so on. The meaning of the message is partly specified by protocol, not guessed statistically from prose.

The paper’s example is socially awkward enough to be memorable. Suppose one agent wants to know when a person arrives at a university because the user wants to avoid that person. If the agent asks the host’s agent, “Do you know when the guest arrives?”, the intended action may be: check your existing information and tell me if you already know. It should not mean: contact the guest, mention my user’s request, and ask them directly. Natural language can blur that distinction. A formal protocol can separate “query your records” from “initiate a new external request.”

In business workflows, this problem appears everywhere. “Check whether procurement approved this vendor” is not the same as “ask procurement to approve this vendor.” “Find whether legal has reviewed the clause” is not the same as “send legal a revised clause.” “Confirm whether the customer accepted the quote” is not the same as “prompt the customer to accept.”

A multi-agent system using unconstrained natural language may coordinate beautifully in demos and misfire in production because agents infer action scope differently. The paper’s practical implication is not that enterprises must resurrect old agent communication languages exactly. It is that agent-to-agent communication needs typed messages, permissions, and auditable semantics.

The adult-supervision version is boring but effective:

Yes, this is less glamorous than “a swarm of agents talking to each other.” That is the point. Production systems should be slightly less glamorous and substantially less ridiculous.

Incentives and mechanisms: design the environment, not just the agent

The paper then moves from individual cognition to collective behavior. This is where many agentic-AI demos become structurally naïve. They assume that if each agent is smart enough, the system will coordinate well. AAMAS research says: no, coordination often depends on the mechanism.

Mechanism design asks how to structure rules, rewards, penalties, and information flows so that agents pursuing their own objectives produce desirable system-level outcomes. The paper uses logistics as an example. Imagine agents representing trucks, each with different capacity, location, and routes. Packages appear dynamically. Instead of one central planner micromanaging everything, an auction mechanism can let trucks bid based on their cost to deliver each package. The lowest-cost agent wins.

The important insight is that intelligence is split between the agents and the mechanism. The agents calculate local costs. The mechanism structures allocation. The system can then optimize different objectives: average delivery time, slowest delivery time, fuel consumption, or another operational target.

For business agent systems, this is a major correction. Many companies imagine multi-agent AI as a group of specialized assistants exchanging messages until magic happens. But once agents represent different departments, tools, vendors, customers, or objectives, conflict is not an exception. It is the environment.

A sales agent wants speed. A legal agent wants risk reduction. A finance agent wants budget discipline. A customer support agent wants satisfaction. A compliance agent wants procedural safety. If the system has no explicit mechanism for resolving these tensions, it will either rely on hidden prompt hierarchy or produce emergent bargaining in natural language. That is not governance. That is office politics with autocomplete.

Mechanism design suggests a more operational question: what rules make local agent behavior produce acceptable global outcomes?

The paper does not provide a ready-made enterprise mechanism library. Its value is diagnostic: if agents have competing objectives but no explicit interaction rules, the system is not mature. It is just a meeting with faster typing.

Coordination is not multiple chat windows taped together

Multi-agent planning and coordination is another AAMAS area the paper brings back into the agentic-AI conversation. The point is simple: many tasks require agents to share goals, decompose work, monitor dependencies, and adapt when the plan changes.

The paper’s travel example expands naturally here. One agent books flights. Another books the hotel. They must coordinate dates. If the flight agent extends the trip by one day because no return flight is available, the hotel agent may need to extend the booking. If the preferred hotel is unavailable, the system may need to choose another hotel, explain the trade-off, and ask for permission.

This is not solved by putting two LLM agents in a chat and hoping they “collaborate.” Real coordination requires shared models of the plan, dependencies, resources, responsibilities, and change propagation. If one agent changes a task, other affected agents need to know not only the fact of the change but its implications.

The paper also raises a practical design choice: should agents be general-purpose or specialized? Specialized agents can be more efficient, but they require more coordination. General agents reduce coordination overhead but may be less reliable in specific domains. The right answer depends on domain complexity and dynamism.

For business leaders, this is an architecture decision, not a branding decision. A “multi-agent system” is not automatically better than a single-agent system. More agents can mean better modularity, clearer responsibility, and improved domain performance. They can also mean more handoffs, more failure points, and more expensive orchestration.

The useful principle is: make the system complex enough to handle the environment, but not more complex than the environment requires. A three-agent architecture for a task that needs one deterministic workflow and one LLM extraction step is not agentic sophistication. It is a very expensive way to avoid writing a form parser.

Negotiation and argumentation: conflict should be structured, not improvised

The paper separates negotiation from argumentation, and this distinction is useful for business design.

Negotiation is useful when agents represent different parties or interests. A logistics provider and a customer may negotiate pickup timing and price. Earlier pickup may cost more because storage is required; later pickup may cost less but impose inconvenience on the customer. The system needs a structured process for offers, counteroffers, constraints, and trade-offs.

Argumentation is more relevant when agents cooperate but bring different expertise. The paper uses a medical-style example: agents may debate whether an elderly patient should receive surgery by weighing risks, expected quality of life, and alternative treatments. Argumentation works only if agents can produce causal links, support claims, refute claims, and revise positions when new facts emerge.

This is where LLM agents look deceptively strong. They can produce arguments in fluent prose. They can simulate disagreement. They can say “on the other hand” until the meeting dies of natural causes. But structured argumentation is not merely generating pros and cons. It requires explicit relations among claims, evidence, assumptions, rebuttals, and conclusions.

For business uses, that difference matters in high-stakes decisions: credit review, legal interpretation, investment committee support, procurement disputes, compliance exceptions, clinical operations, or risk management. A useful AI system should not only output a recommendation. It should show which claims support the recommendation, which assumptions are uncertain, which counterarguments were considered, and what evidence would change the conclusion.

That is the difference between an agent that “has an opinion” and an agent that participates in a governed decision process.

Norms and institutions: rules are not just constraints pasted onto prompts

The paper’s section on norms and institutions is one of its strongest because it explains why simple guardrails are not enough.

A norm is not merely a rule that everyone follows mechanically. Norms matter precisely because violations sometimes become appropriate. Speeding is normally prohibited. But someone rushing a woman in labor to the hospital may be treated differently. An employee may obey a boss’s task request at work, but not a personal request outside the role boundary. These examples are not edge-case decorations. They show that norms depend on roles, context, values, and exceptions.

Current agentic AI systems often treat social structure as something the foundation model will implicitly learn from data, plus a few manually added guardrails. The paper argues this is fragile. Many normative violations are rare, diverse, and context-dependent. They may not appear often enough in training data for reliable learning. Worse, foundation models are trained disproportionately on data from particular societies, especially Western contexts, so implicit norms may not transfer across cultures or institutions.

For business deployment, this is not abstract ethics. It is operational design. An enterprise agent needs to know the difference between:

• what it can do technically;
• what it is allowed to do legally;
• what it should do under company policy;
• what it may do only with human approval;
• what it should refuse even when a senior user asks;
• what exceptional context changes the normal rule.

This cannot be safely outsourced to vibe-based prompt compliance. Enterprises already understand roles, obligations, permissions, prohibitions, approvals, audit trails, and exception handling. Agentic AI should be embedded into those institutional structures, not floated above them as a magical worker who “understands the company culture.” It probably does not. It read a lot of PDFs. Different achievement.

Trust, reputation, and theory of mind: agents need memory of others, not just memory of tasks

The paper also connects trust, reputation, collective decision-making, game theory, and theory of mind. These mechanisms are related because they all concern how agents reason about other agents.

Most current agentic systems treat interactions as relatively stateless. They may have memory of a task or conversation, but they often lack durable models of other actors’ reliability, incentives, preferences, expertise, and past behavior. In open environments, that is a serious weakness.

Trust and reputation models let agents update expectations over time. A purchasing agent should not treat every vendor equally if some vendors have histories of hidden costs, late delivery, or reliable replacement policies. A customer-facing agent should know whether another internal system has produced accurate data historically. A compliance agent should know which upstream processes are usually trustworthy and which require verification.

Theory of mind goes further. It concerns reasoning about others’ beliefs, goals, intentions, and capacities. The paper emphasizes that humans often use role and context to infer what others likely know and want. If someone is a doctor in a hospital, we assume medical knowledge and a patient-care context. If someone is a car salesperson, we infer different incentives. Prior experience modifies these assumptions.

In agentic AI, this matters because cooperation becomes expensive when every assumption must be verified explicitly. Without some model of what other agents know, intend, or can do, agents must over-communicate, under-trust, or guess. None of these are attractive in production.

For businesses, the practical translation is a persistent counterparty model:

This is not science fiction psychology. It is closer to vendor management, access control, model monitoring, and organizational memory. The paper’s point is that agency becomes social once agents act among other agents. Trust must therefore be modeled, not assumed.

What the paper directly shows, and what Cognaptus infers

Because the paper is conceptual, it is important not to oversell it as empirical proof. It does not test an LLM-agent architecture, compare benchmark performance, or quantify enterprise ROI. Its evidence is comparative and theoretical: it maps established AAMAS concepts against gaps in contemporary agentic AI.

That does not make it weak. It makes its contribution different. The paper provides a framework for asking better design questions before deployment.

Cognaptus’ business inference is straightforward: the next stage of enterprise agent deployment should be less about “how many agents can we chain?” and more about “what governance mechanisms make their autonomy safe, inspectable, and useful?”

That sounds less exciting. It is also how systems survive contact with accounting, legal, customers, regulators, and Monday morning.

A practical design checklist for governed agentic workflows

The paper does not provide a checklist, but its mechanisms imply one. Before deploying agentic AI into a real business workflow, teams should be able to answer these questions:

1. Goal structure: What explicit goal is the agent pursuing, and what trade-offs are forbidden even if they improve a local metric?

2. Belief state: What does the agent currently believe about the world, and which beliefs come from verified systems rather than generated text?

3. Commitment model: What has the agent committed to do, for whom, by when, and under what authority?

4. Action scope: Which actions can the agent take directly, which require another agent, and which require human approval?

5. Communication semantics: Are inter-agent messages typed, or are agents interpreting each other’s prose like overconfident interns in a Slack channel?

6. Coordination dependencies: If one agent changes a plan, which other agents or humans must be notified before downstream actions occur?

7. Incentive mechanism: When agents optimize different objectives, what rule resolves conflict?

8. Norms and exceptions: What obligations, permissions, prohibitions, and role boundaries govern the agent, and how are exceptions handled?

9. Trust memory: How does the system update reliability assessments for agents, tools, vendors, data sources, and human counterparties?

10. Audit and rollback: When the system acts wrongly, can the organization reconstruct the chain of beliefs, messages, commitments, and actions?

This checklist is not a substitute for technical implementation. It is a way to prevent agentic AI strategy from collapsing into “let us add more tools and hope the model behaves.” Hope is not an architecture. It is a mood.

The boundary: this is a design thesis, not a deployment benchmark

The main limitation of the paper is also its editorial strength. It is not trying to win a benchmark. It is trying to correct the conceptual foundation of agentic AI.

That means readers should not take it as evidence that every enterprise must immediately build formal BDI agents, implement FIPA-style protocols, or reconstruct the entire AAMAS literature before deploying a useful assistant. Many business processes can benefit from simpler automation: extraction, summarization, drafting, routing, monitoring, and decision support. Not every workflow needs a full multi-agent institutional model.

The boundary appears when autonomy increases. The more an AI system can act continuously, coordinate with other systems, affect real resources, represent different stakeholders, or make changes that are hard to undo, the more the paper’s mechanisms become relevant. A lightweight drafting assistant can be governed with review. An autonomous procurement, scheduling, trading, or customer-resolution system needs much stronger architecture.

The paper also leaves open the hard engineering question: how exactly should LLM flexibility be combined with formal agent structures in scalable production systems? It points to the direction, not a finished product design. There will be trade-offs between rigidity and adaptability, cost and assurance, specialization and coordination overhead, local intelligence and central control.

But that uncertainty does not weaken the core message. It clarifies where the work begins.

From agent demos to agent institutions

The phrase “agentic AI” has become broad enough to cover everything from a chatbot with tools to a multi-agent workflow with memory and external actions. The paper’s warning is that this broadness hides a missing layer. Agency is not just the ability to act. It is the ability to act coherently, accountably, and socially within a structured environment.

That is why the AAMAS perspective matters. It reminds the current AI industry that agents do not become reliable merely by becoming more fluent. They become reliable when their goals, beliefs, communications, commitments, incentives, norms, trust relations, and institutional roles are made explicit enough to inspect and govern.

The business takeaway is not “stop building AI agents.” It is sharper: stop mistaking a clever LLM loop for an organizational actor. Use LLMs where they are powerful, but wrap them in architectures that make their actions bounded, their reasoning inspectable, their communication auditable, and their autonomy supervised by rules stronger than vibes.

Agentic AI does not need less ambition. It needs adult supervision.

Cognaptus: Automate the Present, Incubate the Future.