When Your Agent Knows It’s Lying: Detecting Tool-Calling Hallucinations from the Inside

The expensive part of an AI agent making things up is not always the sentence it writes. Sometimes it is the API call it sends.

A chatbot can hallucinate a policy clause and embarrass itself. An agent can hallucinate a function call and move money, query the wrong data, calculate the wrong dose, bypass an audit trail, or quietly pretend it used a tool when it actually guessed. That is a different species of failure. The output may still look tidy. The JSON may still parse. The function name may even exist. The problem is that the agent has selected the wrong action in a system that treats actions as real.

The paper Internal Representations as Indicators of Hallucinations in Agent Tool Selection takes that problem seriously.¹ Its central idea is simple but technically interesting: when an LLM is generating a tool call, its final-layer internal representations may already contain enough signal to tell whether the call is likely wrong. Instead of asking the model again, sampling three more answers, or checking the output after the fact, the system trains a lightweight classifier on the model’s own hidden states and uses that classifier as an execution-time gate.

That does not mean the model has a tiny conscience. Calm down. It means the model’s internal activations may encode uncertainty, mismatch, or structural inconsistency before the external system executes the proposed call. For production agents, that is the part worth watching.

Tool-calling hallucination is not just broken JSON

A common misunderstanding is to treat tool-calling failure as a syntax problem. The model emits malformed JSON, uses a function that does not exist, or forgets a closing brace. That happens, but it is the easy version of the problem. Validators already catch much of it.

The paper defines tool-calling hallucination more broadly. A call can fail even when it looks structurally clean.

This broader definition matters because enterprise agents rarely fail only by misspelling a function. They fail by choosing a plausible but inappropriate action. They convert the wrong unit. They call the right calculator with the wrong parameter. They “estimate” instead of invoking a verified system. They route a customer request through a workflow that looks reasonable until someone checks the backend logs.

In other words, tool hallucination is not merely an output-quality issue. It is an action-governance issue.

The mechanism: read the model before the tool fires

The paper’s method starts from a mechanism-first hypothesis: the final transformer layer contains discriminative information about whether the generated tool call is correct or hallucinated.

The pipeline is not complicated, which is part of its appeal.

First, the authors use structured tool-calling examples from the Glaive Function-Calling dataset. The dataset contains multi-domain interactions with reference tool calls and parameters. The paper then builds specialized agents across domains such as calculation, personal finance, health metrics, sustainability, and digital commerce. These are not random domains. They are domains where a wrong tool call can produce a numerically precise but operationally false answer, which is everyone’s favorite kind of disaster.

Second, for each interaction, the system removes the reference call from the prompt, asks the target model to produce the tool call, and compares the model’s predicted call against the reference. If the function name or normalized arguments do not match, the prediction is labeled as hallucinated. If they match, it is labeled as non-hallucinated.

This label-generation process is automatic rather than manually annotated. That is useful for scaling, but it also creates one of the paper’s important boundaries: “does not match the reference” is not always the same as “functionally wrong.” Two calls can differ syntactically yet be equivalent, and a reference call can itself be imperfect. The paper acknowledges this under the broader problem of reference-agreement labeling and function equivalence.

Third, the system extracts internal features from the final layer of the same forward pass that produced the tool call. The paper focuses on representations associated with semantically important regions of the generated call: the function name, the argument span, and the closing delimiter. Those representations are compressed into a feature vector and passed into a small feed-forward classifier. The classifier predicts whether the call is hallucinated.

The practical loop looks like this:

User query
   ↓
LLM generates a proposed tool call
   ↓
Final-layer hidden states are extracted from the same generation pass
   ↓
Lightweight classifier estimates hallucination risk
   ↓
Policy gate decides: execute, block, ask for confirmation, retry, or repair

That last line is where the paper becomes business-relevant. The detector is not just a score. It is a possible control point before an external system is touched.

Why single-pass detection changes the deployment conversation

Most hallucination detection methods are easier to explain than to deploy. Generate multiple outputs. Compare consistency. Ask another model to judge. Retrieve facts. Run semantic similarity. These methods can work, but they add latency, cost, and orchestration complexity. Agents already create enough moving parts; adding a miniature committee meeting before every API call is not always charming.

The paper compares its internal-state method with two black-box consistency-style baselines: Non-Contradiction Probability and semantic similarity. In the adapted setup, both baselines repeatedly sample the agent’s tool calls and compare agreement across outputs. The paper describes these baselines as requiring multiple invocations, while the proposed method uses the same forward pass used for generation.

This is the key trade-off. The paper is not simply saying, “our detector gets the highest score.” In fact, the baseline comparison table reports higher overall accuracy for the multi-sample baselines in several settings. The point is more operational: the internal-state method offers a real-time signal without repeated generation.

For production systems, that difference is not cosmetic. A detector that runs inline can sit directly before execution. It can become part of the agent runtime. A detector that requires repeated sampling behaves more like a diagnostic add-on: useful, but more expensive and harder to apply everywhere.

This distinction should shape how we read the results.

The main evidence: useful signal exists, but it is uneven

The paper evaluates three open-source target models: GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B. Each model gets its own classifier, because internal representations are architecture-specific. That is not a minor implementation detail. It means the detector is not a universal plug-in; it is trained for the model it monitors.

The reported accuracies for the internal-representation approach range roughly from 73% to 86% across the tested models. GPT-OSS-20B performs best overall, with 86% accuracy in the detailed classification table. Llama-3.1-8B reports about 73% accuracy. Qwen-7B reports about 74% accuracy.

The more interesting numbers are class-specific.

This table is where the article’s interpretation should not get lazy. Accuracy alone can flatter a detector when non-hallucinated examples dominate. For GPT-OSS-20B, the detector is precise when it says a call is hallucinated, but its hallucination recall is only 0.53 in the detailed table. That means it is not catching nearly half of the hallucinated calls in that evaluation. A production system cannot treat this as a complete safety layer.

Qwen-7B shows the opposite concern. Its hallucination recall is 0.62, but hallucination precision is only 0.34. That is a lot of false alarms. In a high-risk workflow, false alarms may be acceptable if the policy is “ask for confirmation before execution.” In a high-throughput customer-service workflow, constant unnecessary blocking could quietly murder the user experience. The detector is not just a technical component; it changes workflow economics.

Llama-3.1-8B sits in the middle, with more balanced hallucination and non-hallucination behavior. This is less dramatic, which is usually where the practical engineering conversation begins.

The baseline comparison supports speed, not victory laps

The paper’s baseline comparison is best read as a comparison with prior work and black-box detection styles, not as proof that internal representations dominate every alternative.

The multi-sample baselines report high precision and high overall accuracy, especially because consistency checks can be strict: if repeated calls agree, the output is treated as more reliable. But these methods require multiple generations. The proposed method gives up some scorecard advantage in exchange for single-pass operation.

That exchange matters. A bank, insurer, healthcare platform, or procurement workflow may care less about leaderboard elegance than about where the guardrail can be inserted.

A useful way to frame the comparison is:

That is the sober reading. The paper’s contribution is not “we beat all hallucination detection.” It is “there is enough signal inside the model to build a cheap execution-time detector.” The latter is narrower, but also more useful.

The ablation says simplicity may be enough

The paper includes an ablation study on feature extraction methods using Qwen-7B. It compares ten ways to aggregate hidden states into fixed-size classifier inputs: last-layer representations, statistical pooling, attention-weighted pooling, last token, first-last-mean, CLS token, multi-scale pooling, max pooling, and min pooling.

The performance differences are small. Last-layer representations perform best in the reported table, with accuracy around 0.746 and AUC around 0.721. Statistical and attention-weighted methods are close behind. More complex aggregation does not materially improve performance.

This is an ablation result, not a second thesis. Its purpose is not to claim that mean pooling is now the sacred ritual of agent safety. Its purpose is to show that the hallucination signal is not so fragile that it only appears under a carefully engineered representation trick.

That is actually encouraging. If simple final-layer features work about as well as heavier variants, then the detector is easier to implement and maintain. Complexity is expensive twice: first when you build it, then when someone has to explain it after an incident review.

The ablation also hints that the signal may be distributed across the generated call rather than concentrated in a single token. Single-token methods do not clearly dominate. That makes intuitive sense. A tool-call hallucination can live in the relationship between the user query, the selected function, and the arguments. The model’s “wrongness” is not always sitting politely on the final token waiting to be inspected.

What this would look like inside an enterprise agent

Cognaptus’ business interpretation is not that every company should immediately demand access to transformer hidden states and start training tiny MLPs next Monday. That would be the usual enterprise AI ritual: read one paper, add three acronyms, make procurement cry.

The more useful lesson is architectural. A production agent should not treat tool generation and tool execution as one indivisible step. There should be a risk gate between “the model proposed this action” and “the system executed this action.”

The paper suggests one possible signal for that gate: internal-state hallucination risk.

A practical enterprise architecture could use several layers:

The detector from the paper fits in the middle. It should not replace schema validation or external verification. It adds a model-native warning signal that other layers cannot see.

For a finance agent, a high-risk tool call could require human confirmation before submitting a transaction or generating a portfolio recommendation. For a healthcare assistant, it could force the system to rerun a calculation or defer to a validated clinical workflow. For an e-commerce agent, it could prevent incorrect discount, refund, or inventory actions. For sustainability reporting, it could flag suspicious emissions calculations before they enter a report that some poor analyst will later pretend was “strategic.”

The value is not mystical interpretability. The value is cheaper diagnosis before damage.

The ROI is in selective friction

A hallucination detector creates value only if the action policy is well designed. Blocking every suspicious call is safe in the same way turning off the server is secure. Technically true; commercially unhelpful.

The operational question is: what should the system do when the detector raises risk?

There are several options.

For low-risk workflows, the agent can retry with a stricter tool-use prompt or ask the user for missing information. For medium-risk workflows, it can request confirmation: “I am about to use the loan calculator with these parameters. Please confirm.” For high-risk workflows, it can escalate to a human reviewer or require an external deterministic check.

The detector’s score should not be treated as a verdict. It should be treated as a routing signal.

This is where false positives and false negatives become business parameters. A false positive adds friction. A false negative allows a bad call through. The acceptable trade-off depends on the workflow. A refund bot, trading assistant, compliance copilot, and calorie calculator do not have the same tolerance for delay or error. If they do, someone has confused “AI strategy” with wallpaper.

The paper’s results are strong enough to justify experimentation, but not strong enough to justify blind automation in high-stakes systems. That is not a weakness; it is the normal state of useful research.

Boundaries that matter for implementation

The paper is careful enough to leave several practical boundaries visible.

First, the method requires access to internal representations. That rules out many closed commercial models unless the provider exposes hidden states or offers a similar monitoring interface. Black-box API users cannot directly implement this exact method. They can imitate the architecture with external classifiers or consistency checks, but not the internal-state detector itself.

Second, the classifier is model-specific. GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B each require dedicated classifiers with different input dimensions. If a company changes its base model, fine-tunes it, quantizes it, or swaps inference infrastructure, the detector may need recalibration or retraining.

Third, the labels depend on reference-call agreement. This is scalable, but imperfect. In real business workflows, there may be multiple valid tools, equivalent argument forms, or context-dependent choices that are not captured by exact matching after canonicalization. A detector trained on rigid reference agreement may learn “different from expected” rather than “operationally unsafe.”

Fourth, the experiments are conducted on structured tool-calling datasets and selected domains. Finance, health, sustainability, commerce, and calculation tasks are relevant, but they are not the entire enterprise universe. Multi-step workflows, changing APIs, ambiguous user intent, authentication contexts, and messy legacy systems can all change the error surface.

Finally, the detector is binary. The paper frames hallucination detection as hallucinated versus non-hallucinated. That is operationally simple, but production systems often need richer diagnosis: wrong tool, wrong parameter, missing context, unauthorized action, stale data, or tool bypass. A binary gate is a useful start. It is not a complete incident taxonomy.

What the paper directly shows, and what we infer

Here is the clean separation.

The paper directly shows that final-layer internal representations from several tested open-source LLMs contain usable signal for detecting hallucinated tool calls. It also shows that a lightweight classifier can use this signal in a single-pass setting, with reported accuracy around 73% to 86% depending on the model. Its ablation suggests that simple final-layer feature aggregation can work about as well as more complex alternatives.

Cognaptus infers that this method is best understood as an execution-time risk gate for LLM agents. Its business value is not merely higher accuracy. It is the ability to place a cheap warning signal between proposed action and external execution.

What remains uncertain is how well this approach transfers to closed models, heavily customized enterprise tools, multi-step agents, noisy reference labels, and workflows where several tool calls may be equally valid. The method is promising, but it is not a substitute for schema checks, policy controls, deterministic validation, logging, and human escalation where stakes demand it.

That distinction is important because agent safety will not be solved by a single clever classifier. It will be solved by layered controls that know exactly what they are controlling.

The quiet shift: from output checking to action governance

The most useful part of this paper is not the phrase “internal representations.” It is the shift in where detection happens.

Traditional hallucination detection often evaluates the answer after it exists. This paper asks whether we can detect trouble while the agent is forming the action. That is a small timing difference with large engineering consequences.

If an agent writes a bad paragraph, the system can edit it. If an agent sends a bad tool call, the system may already have changed the world. So the detector has to move upstream.

The paper does not prove that agents know they are lying in any human sense. It shows something more useful and less theatrical: the model’s own hidden states may contain enough evidence to warn us before the tool fires.

That is the kind of finding enterprise AI needs more of. Not another promise that agents will become perfectly reliable after the next benchmark. Not another dashboard decorated with confidence scores of mysterious ancestry. A concrete place to insert friction before automation turns a plausible guess into an executed operation.

The agent may not know it is lying.

But the system might still learn when not to trust it.

Cognaptus: Automate the Present, Incubate the Future.