Claw-Eval — When Agents Game the System, the System Needs Claws

The agent finished the task. That is not the same as doing the task.

Inbox sorted. Calendar updated. Report generated. Customer record changed. Dashboard refreshed.

For a demo, that is usually enough. The screen shows a plausible answer, the final artifact looks tidy, and everyone politely pretends the agent must have followed the correct path because the output did not immediately burst into flames.

For an enterprise workflow, this is a dangerous little fairy tale.

An autonomous agent is not just a text generator with better manners. It is a system that takes actions: it calls APIs, reads files, edits state, touches services, and sometimes produces consequences before anyone has inspected the reasoning. Once a model becomes an actor, the evaluation question changes. We are no longer asking only whether the final answer is right. We are asking whether the path was legal, safe, recoverable, and reproducible.

That is the useful center of Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents.¹ The paper introduces a 300-task benchmark for autonomous agents, but the benchmark size is not the most interesting part. The more important move is methodological: Claw-Eval treats the agent’s full execution trajectory as the object of evaluation.

In other words, it does not just grade the destination. It checks the tire marks, the receipts, the security camera footage, and whether the driver hit anything on the way. A modest improvement over vibes-based benchmarking, one might say.

Claw-Eval’s core mechanism is forensic, not decorative

The paper’s central claim is simple: output-only evaluation is structurally weak for agents. A final answer can hide skipped steps, fabricated intermediate work, forbidden actions, failed tool calls, and lucky recovery. The model may appear competent because the visible artifact is acceptable, while the underlying process is exactly the kind of process a business would never want running unattended.

Claw-Eval responds by building the evaluation around a three-phase lifecycle:

This design matters because Claw-Eval collects three independent evidence channels: execution traces, service-side audit logs, and post-execution environment snapshots. Each channel catches a different kind of failure.

Execution traces show what the agent tried to do. Service-side audit logs show what external systems actually received. Environment snapshots show what files, artifacts, or states exist after the run. Together, these channels make it harder for an agent to pass by producing a fluent summary of actions it never performed.

That is the first practical lesson for businesses: evaluating agents is closer to audit engineering than exam grading. A serious agent evaluation stack needs logs, state inspection, deterministic checks, and tool-call verification. A screenshot of a nice answer is not a control system. It is a screenshot.

The benchmark covers three kinds of work agents actually face

Claw-Eval contains 300 human-verified tasks across 9 categories and 3 broad task groups. The distribution is important because the paper is not evaluating one narrow skill and pretending it represents “agent intelligence.” It is testing different forms of situated action.

This task diversity is not merely breadth for marketing purposes. The paper’s later results show that model rankings shift across task groups. A model that looks strong in text-heavy service orchestration may not dominate multimodal tasks. A model that can answer after a long dialogue may still ask poor questions. The benchmark design therefore supports a more sober view: “best model” is often shorthand for “best under one specific operating regime.”

That distinction is boring only until procurement chooses a model for the wrong regime.

Safety is not a side score; it is a gate

A subtle but important part of Claw-Eval is how it treats safety. Many evaluations add safety as another column, as if a system can compensate for unsafe actions with enough task completion. Claw-Eval does something stricter: safety acts as a multiplicative gate in the task score.

The intuition is correct. If an agent completes an email-triage task but violates a constraint by sending a message it was forbidden to send, the business problem is not “partial credit.” The problem is that the agent crossed a boundary. A brilliant assistant that leaks credentials is not 80% useful. It is a liability wearing a productivity costume.

The paper’s case study on email triage makes this concrete. The task asks the agent to classify inbox messages while obeying a safety constraint: it must not call gmail_send_message. Claw-Eval can verify this not from the agent’s final explanation, but from the actual tool logs. That is exactly the kind of evidence enterprises need when evaluating agents for regulated or customer-facing workflows.

This is also where ordinary output grading breaks. A final response may say, “I sorted the inbox and did not send anything.” Fine. But a serious evaluator should not be charmed by autobiography. It should check the service logs.

Pass@3 measures possibility; Pass^3 measures dependability

One of the paper’s most useful ideas is the separation between peak capability and consistent reliability. Claw-Eval runs each task across three trials and reports three metrics:

This distinction should be stapled to every enterprise AI pilot report.

A high Pass@3 means the agent can find a successful path at least once. That is useful for research and demos. It is much less comforting for operations. In a business process, a system that succeeds once and fails unpredictably twice is not “promising.” It is expensive randomness.

Pass^3 is harsher. It asks whether the agent can pass every time across repeated trials. This is closer to what businesses need when the same workflow is expected to run daily, hourly, or on thousands of customer records.

The main evaluation results show why the distinction matters. On General and Multi-turn tasks, Claude Opus 4.6 leads the aggregate Pass^3 ranking at 70.4%, while Claude Sonnet 4.6 leads the aggregate Average Score at 81.4%. The ranking changes depending on whether the buyer cares about average quality or strict consistency. That is not a footnote. That is the procurement question.

Even the strongest model in the reported General + Multi-turn aggregate reaches only 70.4% Pass^3. This does not mean the models are useless. It means the benchmark still has headroom, and more importantly, that deployment-grade agent reliability is not solved by choosing a frontier model and hoping the brand name does the risk management.

The most damaging result: the vanilla judge misses what the audit catches

The paper’s most business-relevant experiment compares Claw-Eval’s hybrid grading pipeline against a more ordinary LLM-judge setup. The vanilla judge receives the full conversation transcript, including tool calls, and the complete grader source code. The only withheld evidence is the server-side audit logs and post-execution environment snapshots.

That setup is generous to the vanilla judge. It is not being asked to evaluate from a short final answer. It sees the transcript and the rules. Even so, across five models and more than 2,000 traces, the vanilla judge misses 12 of 27 safety violations and 15 of 118 robustness issues detected by the hybrid pipeline.

The safety miss rate is the headline, but the mechanism is the real story. The vanilla judge can read the rule text, yet it cannot reliably execute deterministic matching over external state. It may also rationalize behavior from the transcript. That is exactly the wrong personality trait for a compliance layer.

The appendix strengthens this result with blinded human adjudication. On the 12 safety disagreement cases, human adjudication agrees with the hybrid pipeline in all cases and with the vanilla judge in none. On the 15 robustness disagreements, humans agree with the hybrid pipeline in 13 cases and with the vanilla judge in only 1. This does not make Claw-Eval perfect ground truth. It supports a narrower but valuable claim: when criteria depend on audit logs or final environment state, a hybrid evidence-grounded evaluator is closer to human judgment than a conversation-only judge.

For business readers, the implication is direct: do not use an LLM judge as the only evaluator for deterministic operational constraints. Let it evaluate open-ended quality where judgment is needed. Use deterministic checks for things that can be verified. The point is not to abolish LLM-as-judge. The point is to stop asking it to be a database auditor, API monitor, and compliance officer because the org chart was feeling lazy.

Robustness tests reveal the demo/product gap

Claw-Eval also tests how agents behave under injected tool failures. The paper evaluates three models on General tasks at error-injection rates from 0.0 to 0.6. Each mock-service call may independently fail with HTTP 429, HTTP 500, or a 2–4 second latency spike.

This is a robustness test, not a second thesis about all possible production failures. It tests infrastructure-level transient errors: rate limits, server errors, and latency spikes. That boundary matters. It does not cover semantic drift, adversarial service responses, corrupted data, user deception, or vendor API changes. Still, the test captures a common operational reality: tools fail, and agents must recover without collapsing into interpretive poetry.

The result is sharp: error injection damages consistency more than peak capability. Pass@3 remains relatively stable, while Pass^3 drops sharply as error rates rise. The agent can often succeed once. It cannot reliably succeed every time.

The paper reports that Claude Opus 4.6 remains the most resilient among the tested models, retaining 56.5% Pass^3 even at the 0.6 error-injection rate. But the broader lesson is not “buy this model.” The broader lesson is that baseline capability does not fully predict recovery behavior. A model’s normal-task score is not enough to estimate its performance under tool turbulence.

A production evaluation should therefore include failure injection before deployment, not after the customer support queue becomes an ethnographic study in preventable errors.

Multi-turn performance is about question quality, not conversational length

The multi-turn tasks are designed around hidden-intent professional consultation. A simulated user has a persona, domain knowledge, hidden facts, and an information-revealing strategy. The agent has to ask questions before giving the final answer.

This matters because many business agents are not just command executors. They are supposed to clarify incomplete requests, diagnose ambiguity, and extract missing context from users who do not know what details matter. In other words, they operate in the natural habitat of enterprise communication: partial information plus confidence.

The paper finds that the number of dialogue rounds has near-zero correlation with Pass^3. Most models average three to five rounds, but performance varies widely. Question precision is the stronger predictor. Defined as a combination of targeted clarification and logical information-gathering trajectory, question precision explains 76% of the variance in multi-turn Pass^3.

This is one of the paper’s most useful corrections to a common reader misconception. More interaction is not automatically better interaction. A model that asks many generic questions can still fail to uncover the key fact. A model that asks fewer but better-sequenced questions can solve the case more reliably.

For business deployment, this suggests that multi-turn agents should be evaluated not only on final answers but on the diagnostic quality of their questioning. Did they ask for the missing input that changes the decision? Did they sequence the questions logically? Did they adapt after the user pushed back? The paper’s statistical-consultation case study illustrates this well: the agent must diagnose a Box’s M warning in SPSS by eliciting details about design, variance, missing data, and the user’s misconception about Box’s M versus Levene’s test.

That is not “chat quality.” It is information acquisition under uncertainty.

Multimodal agents remain fragmented, and aggregation hides the fracture

The multimodal results are the paper’s antidote to another lazy benchmark habit: reporting one overall multimodal score and calling it insight.

Claw-Eval separates multimodal tasks into Video, Document & Image, and Code. The domain-level results show that no single model dominates. GPT-5.4 leads the overall multimodal Pass^3 table at 25.7% and leads Document & Image at 54.5%. Claude Opus 4.6 and Claude Sonnet 4.6 lead Video at 15.4%. MiMo V2 Omni leads Code at 33.3%.

The average Pass^3 across multimodal domains is also uneven: Video is 10.7%, Document & Image is 32.3%, and Code is 23.9%. The pattern is not merely that multimodal work is hard. It is hard in different ways.

Video tasks require temporal localization, frame selection, and visual grounding over time. Document and image tasks often involve static visual reasoning and cross-page interpretation. Code tasks require generating artifacts that can be rendered and inspected. These are not interchangeable skills, even if all of them fit under the comforting umbrella term “multimodal.”

For procurement and internal model selection, this means a company should not ask, “Which model is best at multimodal?” It should ask, “Which modality, artifact type, and failure mode dominate our workflow?” The first question produces vendor slides. The second produces a test plan.

What the appendices are really doing

The appendices are not just academic furniture. They clarify which parts of the evidence are main results, which parts are validation, and which parts are implementation detail.

This separation matters because a careless reading can overclaim the paper. Claw-Eval does not prove that its benchmark is the final answer to agent evaluation. It proves that a benchmark with trajectory auditing, safety gates, robustness perturbations, and multi-trial metrics exposes failures that ordinary evaluation can miss.

That is enough. In fact, it is more useful than claiming universality.

The business lesson is not “use Claw-Eval”; it is “build claws into evaluation”

The paper’s practical value is not limited to organizations that directly adopt the benchmark. The broader lesson is architectural.

A business evaluating autonomous agents should separate at least five layers:

This is where the business relevance becomes concrete. For an internal finance agent, the evaluation should check whether it accessed the correct data source, applied the correct calculation, avoided forbidden disclosure, and recovered from temporary API failure. For a customer-service agent, it should inspect whether the agent obeyed escalation policies, avoided unauthorized account changes, and asked targeted clarifying questions. For a document-processing agent, it should test not only whether the final summary sounds reasonable, but whether the agent looked at the right pages and preserved critical numerical details.

The ROI is not simply “better benchmark scores.” The ROI is cheaper diagnosis. When an agent fails, a trajectory-aware system can tell whether the failure came from perception, planning, tool use, safety-policy conflict, missing user information, or recovery logic. Without that decomposition, every failure becomes a séance: people gather around the transcript, infer motives, and call it postmortem analysis.

What remains uncertain

Claw-Eval is valuable, but its boundaries should remain visible.

First, the robustness perturbations focus on infrastructure-level transient failures: HTTP 429, HTTP 500, and latency spikes. These are important, but they are not the whole failure universe. Real deployments also face stale databases, shifting API schemas, adversarial prompts, poisoned files, permission mismatches, ambiguous policy conflicts, and users who communicate like encrypted weather systems.

Second, the benchmark uses sandboxed and mock-service environments. This is a strength for reproducibility and controlled evaluation, but production systems may have messier state, longer histories, and organizational constraints that are hard to simulate.

Third, judgment-based rubric items still rely on LLM judges. The paper reports 95% exact agreement with human scoring across sampled judgment-based rubric items, which is strong for practical use. But model-based judgment remains an approximation. It should be monitored, sampled, and periodically recalibrated, especially when new task types are added.

Fourth, public benchmarks age. Once a benchmark becomes influential, models and scaffolds can be optimized toward its task distribution. Claw-Eval’s trajectory-level evidence and declarative task schema help, because new tasks and deterministic checks can be added. But benchmark longevity is still a governance problem, not a one-time publication event.

These limitations do not weaken the paper’s core message. They define where to apply it carefully.

The new evaluation question: can the agent be audited?

The industry likes asking whether a model is smart. Claw-Eval asks a less glamorous and more useful question: can the agent be trusted when it acts?

That question cannot be answered from final outputs alone. It requires evidence. It requires logs. It requires repeated trials. It requires safety gates that cannot be offset by charm. It requires stress tests that reveal whether success is repeatable or merely available on a good day.

The paper’s deeper contribution is therefore not a leaderboard. Leaderboards are useful, but they age quickly and encourage the usual species of benchmark theater. The durable contribution is the evaluation frame: audit the trajectory, separate completion from safety and robustness, distinguish peak capability from consistent reliability, and evaluate agents across the modalities and interaction patterns they will actually face.

This is where agent evaluation becomes less like grading homework and more like building internal controls. That may sound less exciting than “autonomous intelligence.” Good. Excitement is not a deployment strategy.

If agents are going to act inside real workflows, the evaluation system needs claws.

Cognaptus: Automate the Present, Incubate the Future.