The Agent's Second Record: What Its Tools Actually Execute

TL;DR for operators

An enterprise agent can invoke a legitimate-looking tool while its underlying process accesses credentials, launches additional commands, or transmits data. The conversation transcript may never reveal those side effects. But monitoring execution alone can also miss attacks that manipulate what the agent reads or decides.

The Agent Cross-Layer Evidence (ACE) benchmark shows why the distinction matters. For silent exfiltration through logging, application evidence barely distinguishes malicious sessions (AUROC 0.586), while operating-system evidence performs strongly (0.976). For retrieval-reference injection, the relationship reverses: application evidence reaches 0.906 versus 0.703 for kernel evidence. These are mean in-distribution results across 10 learned detectors. Combining both evidence sources leads all 10 detectors in-distribution and seven under held-out-attack-family evaluation, although kernel-only evidence wins for some detectors and attack mechanics.

For security architects, the implication is to correlate agent interactions with filesystem, process, and network activity rather than assume either monitoring layer is sufficient. Detector selection, runtime portability, and alert thresholds still require validation. ACE evaluates completed sessions under controlled conditions; it does not establish real-time prevention, acceptable production false-positive rates, or universal attack coverage.

A legitimate tool call can conceal unauthorized execution

Consider an enterprise agent invoking a tool to retrieve information. The transcript records the expected request and response. The tool’s advertised interface appears legitimate. Yet the process serving that request may also open a credential file or initiate an unauthorized network connection.

Neither the tool description nor the conversation necessarily records those side effects.

The reverse problem also occurs. An attacker can manipulate retrieved references or influence the agent’s subsequent choices without generating unusual operating-system activity. Monitoring process behavior alone may then provide little evidence of the compromise.

These cases make security monitoring an evidence-selection problem. Before choosing a classifier, an operator must determine whether the relevant attack leaves its strongest observable trace in the application interaction, the underlying execution, or both.

In On the Effectiveness of Kernel-Level Evidence for Agent Security, Spencer King and colleagues investigate this question using paired observations of agent behavior.¹

Their results show that the most informative evidence source changes substantially across attack mechanics.

ACE records what the agent says and what its processes do

The researchers construct ACE from 4,047 controlled agent sessions involving 17 threat models, six attack-delivery families, and 12 observable attack mechanics.

The primary environment runs Claude Haiku 4.5 through the Claude Code CLI inside dedicated Docker containers. Attack fixtures involve 31 real third-party MCP servers and built-in agent tools.

Each session collects two synchronized evidence streams:

• Application evidence: the served MCP tool manifest, user prompt, and agent transcript.
• Kernel evidence: system calls made by the agent’s monitored process tree, including operations involving files, processes, and network connections.

A system call is a program’s request to the operating system. Recording these requests provides evidence of execution behavior that may not appear in the agent’s conversation.

The researchers capture 15 syscall tracepoints using strace. For numerical detectors, they extract 29 application features and 57 kernel features. The combined representation contains 86 features. Language-model detectors instead receive text-rendered evidence, including a bounded summary of the syscall stream.

The benchmark distinguishes an important labeling condition: a malicious fixture is not automatically a confirmed malicious execution.

Of the 4,047 sessions, 1,358 have confirmed observable malicious effects. Canonical evaluation treats 2,293 sessions as negative, including benign, dormant, and signature-free latent cases, while excluding 396 ambiguous signal-bearing latent sessions.

Consequently, the benchmark measures detection of confirmed attack effects under defined observation rules, not whether malicious code or instructions were merely present.

The attack mechanic determines where the evidence appears

ACE’s per-mechanic analysis provides its clearest explanation of why the two observation layers complement each other.

AUROC measures how well a detector ranks malicious sessions above negative sessions across thresholds. A score of 0.5 indicates chance-level discrimination; 1.0 indicates perfect ranking in the evaluated sample.

Selected results from Section 3.5, Table 2 report mean in-distribution AUROC across the 10 learned detectors:

For silent logging exfiltration, application evidence offers little discrimination beyond chance. The unauthorized execution produces much stronger kernel-visible evidence.

Retrieval-reference injection reverses that relationship. Its manipulation is more informative in the application’s interaction records than in filesystem, process, or network behavior.

Preference manipulation demonstrates a third pattern: neither individual layer reaches AUROC 0.75, but combining them reaches 0.834.

Across all 12 attack mechanics, combined evidence exceeds the paper’s 0.75 discrimination threshold. It achieves the highest mean in-distribution AUROC for 11 mechanics, with kernel evidence narrowly leading for silent logging exfiltration.

This supports monitoring both semantic interactions and execution effects. It does not imply that every attack will leave a detectable signature in either stream.

Combined evidence usually helps, but the strongest detector is kernel-only

The researchers next evaluate whether these relationships survive changes in detector architecture and attack distribution.

They compare boosted trees, tabular foundation models, frozen LLM judges, and LoRA-fine-tuned smaller language models. Five stratified folds measure in-distribution performance; seven OWASP-aligned folds test attack families excluded from training or demonstration examples.

Representative held-out-family results from Appendix H.1, Table 8 are:

For XGBoost, combined evidence raises held-out-family AUROC from its best single-view result of 0.750 to 0.850. AdaBoost improves from 0.733 to 0.840.

Yet the strongest overall held-out-family result belongs to the fine-tuned Llama-3.1-8B detector using kernel evidence alone. Its combined-view AUROC is lower, at 0.791.

The implication is model-dependent: more evidence does not guarantee better generalization, even when the added evidence contains useful information.

The authors also test prompting and representation choices. OWASP threat-taxonomy grounding improves the tested zero-shot Qwen configurations, while family-excluded labeled examples further improve their held-out-family discrimination. These are prompting robustness results, not independent confirmation that LLM judges are production-ready security monitors.

Existing security tools provide incomplete coverage

ACE includes comparisons with two conventional security approaches.

Cisco’s mcp-scanner, which inspects MCP manifests, achieves 15.9% recall at a 16.3% false-positive rate on the eligible tampered-MCP cohort. The evaluated learned detectors perform better at their native thresholds.

Manifest inspection cannot observe every harmful operation implemented behind an unchanged tool interface. Moreover, 2,068 ACE sessions use non-MCP delivery vectors and fall outside this scanner’s applicable scope.

The researchers also port the syscall-observable subset of Falco’s default runtime rules to ACE traces. That baseline reaches held-out-family AUROC 0.572. At its native any-rule-fires threshold, it flags every evaluated session, producing both 100% recall and 100% false positives.

This result does not invalidate host-intrusion detection. It shows that this particular default rule configuration poorly distinguishes legitimate and malicious behavior in the tested agent workloads.

Agent processes routinely create files, launch subprocesses, and communicate over networks. Operational detection must distinguish authorized from unauthorized instances of those activities, not merely identify that they occurred.

Runtime portability and alert calibration remain deployment decisions

A separate experiment tests 592 sessions using Devstral 2 123B and a Python ReAct scaffold.

Three representative detector families retain mean AUROC above 0.84. TabPFN, however, falls to 0.662.

The authors associate its deterioration with runtime-dependent filesystem behavior. One feature, path_n_total, is nonzero throughout the primary ACE corpus but zero in 57% of the alternate-runtime sessions.

A detector can therefore learn properties of its original execution environment that are not reliable indicators of attacks elsewhere.

This transfer experiment uses conditioning data from the same attack families and changes both agent model and scaffold. It should not be interpreted as simultaneous generalization to an unseen runtime and unseen attack families.

Identifier retention introduces another operational choice. When both training and evaluation use redacted kernel summaries, the Llama-3.1-8B detector’s held-out-family AUROC falls from 0.922 to 0.759. Numerical detectors nevertheless retain substantial discrimination without literal pathnames or IP addresses.

Security teams should evaluate privacy-preserving representations separately rather than assuming that removing identifiers has no detection cost.

Finally, AUROC does not determine how many alerts an operational team must investigate. On the eligible tampered-MCP subset, Qwen3-80B achieves 73.2% recall at its native 7.3% false-positive rate, but only 0.7% recall under the paper’s strict 1% false-positive ceiling. These thresholds are evaluated on benchmark data, not calibrated against an independent deployment population.

Better retrospective evidence is not automatic prevention

ACE provides substantial evidence for comparing detection signals under controlled conditions. Its paired sessions, held-out attack families, multiple model families, and ablations strengthen that comparison.

Three limits materially affect deployment interpretation.

First, classification occurs after a completed session. The experiments do not demonstrate that a detector can intervene before credential access, data transmission, or another harmful operation finishes.

Second, the two monitored layers have coverage gaps. Content-only manipulation, some timing channels, and whitespace steganography can evade both when they leave no captured signature. The served tool manifest is also recorded only once per session.

Third, container-bounded process attribution does not directly establish reliable attribution in shared hosts, long-running daemons, embedded agents, or multi-agent workflows. The controlled network sinkhole can also produce execution patterns different from production traffic.

These limitations prevent treating ACE’s reported AUROC values as estimates of real-world prevention effectiveness.

What security architects should change

For organizations deploying tool-using agents, the evidence supports three connected decisions: preserve application and kernel telemetry together, maintain reliable attribution to the responsible agent session, and validate detection models against representative legitimate and malicious activity in each target runtime.

Cognaptus infers that this favors cross-layer observability as part of defense in depth, alongside tool permissions, sandboxing, and enforcement controls. The paper directly evaluates detection, not the effectiveness of those additional controls.

ACE’s contribution is to demonstrate that an agent’s visible interaction and its executed behavior carry different security information. Reliable monitoring requires access to both, with model selection and alert thresholds tested against the environment in which decisions will actually be made.

Cognaptus: Automate the Present, Incubate the Future.