Judge, Jury, and GPT: Bringing Courtroom Rigor to Business Automation

TL;DR for operators

A web agent that looks impressive in a demo may still fail when asked to complete ordinary live tasks across messy websites. That is the central finding of An Illusion of Progress? Assessing the Current State of Web Agents, which introduces Online-Mind2Web, a benchmark of 300 realistic tasks across 136 websites.¹

The numbers are not subtle. OpenAI Operator reaches 61% success. Claude Computer Use 3.7 reaches 56.3%. Most other evaluated agents cluster around 30%. This is far below the near-90% performance reported in some earlier WebVoyager-style evaluations. The gap is not just about model capability. It is also about measurement quality.

The paper’s practical lesson for business automation is simple: the evaluator is part of the system. If the judge is weak, the agent looks stronger than it is. If the benchmark allows shortcuts, the agent learns to look productive without proving that it can follow the real workflow. If the evaluation only checks the final answer, it misses whether the agent applied the right filters, visited the correct page, submitted the required form, or quietly hallucinated a result after getting lost. Very human. Not very useful.

The authors propose WebJudge, an LLM-based evaluator that uses task requirements, action history, and selected intermediate screenshots to judge whether an agent actually completed a task. With o4-mini, WebJudge reaches 85.7% agreement with human judgement; WebJudge-7B reaches 87% agreement while reducing evaluation calls. That does not remove the need for human audit, but it makes scalable inspection more plausible.

For Cognaptus clients, the point is not “web agents are useless.” The point is sharper: automation should be deployed with courtroom-style evidence. Each workflow needs a defined task, explicit success criteria, traceable behaviour, and a judge that checks whether the process was completed correctly. The agent is only half the product. The other half is proof.

The courtroom problem in business automation

Imagine an accounts-payable agent that receives an invoice, checks the supplier, approves the payment, and emails Finance. In a demo, it may appear to work because the final screen says “approved.” A human reviewer, however, would ask less glamorous questions.

Was the invoice amount read correctly? Was the supplier matched against the right vendor record? Was the approval threshold applied? Was the confirmation email actually sent, or merely drafted? Did the agent act on the current invoice, or on a stale page from the previous run?

That is the difference between completion theatre and operational evidence.

Web agents face the same problem. A task such as “find the lowest-priced 27–32 inch Samsung or LG 4K IPS monitor” is not solved by typing the whole instruction into a search bar and reporting something that looks plausible. It requires constraints to be applied correctly: brand, size, display type, resolution, price ordering. Miss one, and the answer may still read confidently while being wrong. Lovely tone. Terrible work product.

The paper matters because it treats web-agent evaluation less like a scoreboard and more like a legal proceeding. The question is not just, “Did the final answer sound right?” It is, “Does the trajectory prove that the task requirements were satisfied?”

That distinction is exactly where business automation either becomes reliable infrastructure or expensive theatre with nicer fonts.

The headline result: web agents look weaker on live, realistic tasks

The paper’s first contribution is Online-Mind2Web, a new benchmark designed to reduce the gap between agent evaluation and how real users actually ask agents to operate on the web.

Earlier benchmarks have different compromises. Offline benchmarks use cached websites, which makes experiments easier but restricts exploration. Sandbox environments allow interaction but cover a limited set of simulated sites. Online benchmarks use live websites, but some have narrow website coverage, simplified tasks, or unreliable automatic evaluation.

Online-Mind2Web tries to make the test more realistic. It contains 300 tasks across 136 websites. The tasks span everyday domains including shopping, housing, finance, health, travel, entertainment, technology, education, government services, jobs, and more. The authors classify tasks by difficulty using reference length, meaning the number of steps a human annotator needs to complete the task. The final benchmark includes 83 easy tasks, 143 medium tasks, and 74 hard tasks.

The result is a benchmark that exposes a less flattering picture of current agent capability.

The most useful interpretation is not that web agents have made no progress. Operator and Claude Computer Use 3.7 are materially stronger than the rest of the field in this study. The useful interpretation is that broad autonomy remains uneven. It is especially fragile when the task requires sustained navigation, multiple constraints, sorting, filtering, form submission, or exact numerical and temporal conditions.

That maps directly to business workflows. Many enterprise tasks are not hard because each step is intellectually deep. They are hard because the sequence is unforgiving. Read this value. Apply this policy. Choose this record, not the adjacent one. Click submit. Capture confirmation. Escalate only if the exception condition is met. Repeat without inventing a shortcut.

Current agents are improving at the first half of that sentence. They are still unreliable at the second.

Why older scores inflated confidence

The paper is careful not to accuse prior benchmarks of being useless. Instead, it shows why their design can inflate confidence.

The authors pay special attention to WebVoyager, a widely used benchmark for online web agents. Its appeal is obvious: it evaluates agents on live websites, which sounds realistic. But the paper identifies three important weaknesses.

First, WebVoyager covers only 15 websites. That limits the diversity of site structures, interaction patterns, and task types. Second, many of its tasks can be solved through shortcut behaviour. The authors build a naive search agent that generates a Google query, clicks a result, and checks whether the answer is present. On a sample of WebVoyager tasks, this simple agent solves up to 51%. On Online-Mind2Web, the same kind of shortcutting works much less well: the search agent solves 22% overall, with success falling to roughly 3% on hard tasks.

Third, automatic evaluation can be unreliable. If the judge accepts a confident final response without checking the actual trajectory, hallucination becomes a performance booster. The agent does not have to complete the task; it has to sound as if it did. A marvellous feature, assuming your QA strategy was written by a casino.

The authors also test the Google Search constraint directly. They restrict agents from using Google Search during main evaluation to focus on navigation and avoid unfair comparisons across different destination websites. In an appendix robustness check, Browser Use improves only modestly when search is allowed, from 26% to 31%. That supports the benchmark’s intended purpose: the problem is not merely that search was disabled. The tasks still require real interaction.

The business translation is straightforward. If a vendor claims high success on a benchmark, the next question should be, “What kinds of failures does this benchmark allow?” A benchmark that permits shortcutting, weak final-answer judging, or narrow website coverage may still be useful for research iteration. It is not sufficient evidence for production readiness.

WebJudge changes the unit of evidence

The paper’s second contribution is WebJudge, an automatic evaluator designed to approximate human judgement more reliably.

The key design choice is that WebJudge does not judge only the final response. It uses three stages.

First, it identifies key points from the task. These are explicit requirements that must be satisfied: filters, ranges, location constraints, sorting instructions, submission requirements, and other conditions.

Second, it selects key screenshots from the trajectory. This matters because web-agent trajectories can be long. Feeding every screenshot into a model creates token overload and makes the judge worse. Feeding only the final screenshot loses crucial evidence. WebJudge tries to preserve the intermediate states that actually prove whether the agent completed the task.

Third, it judges the outcome using the task description, key points, selected screenshots, and action history.

This is a more realistic evidentiary standard. In business terms, it resembles an audit trail. The agent’s final claim is not enough. The system needs evidence that the correct conditions were applied during execution.

The results are strong enough to be operationally interesting. WebJudge powered by o4-mini reaches 85.7% agreement with human evaluation and a 3.8% average success-rate gap. WebJudge-7B reaches 87% agreement with a 3.9% gap, while reducing evaluation to a fixed two calls per trajectory. GPT-4o-based WebJudge also outperforms several existing automatic evaluation methods, though o4-mini and WebJudge-7B are the headline variants in the paper’s main discussion.

The comparison against existing evaluators explains why this matters. WebVoyager-style evaluation can suffer from token overload because it attempts to include many screenshots. Final-screenshot evaluation can miss important intermediate evidence. AgentTrek-style evaluation, in the authors’ empirical comparison, shows lower agreement with human judgement on the evaluated agents. WebJudge sits between too little evidence and too much evidence: it filters for the evidence that matters.

That is the right direction for business automation.

An invoice workflow does not need every screen recording frame to be analysed forever. It needs the key proof points: invoice ID, amount, supplier, approval rule, action taken, confirmation generated, exception handling. The evaluator should be strict about those points and indifferent to irrelevant UI noise.

The lesson is not “use an LLM as judge and relax.” The lesson is “make the judge inspect the right artefacts.” A bad LLM judge is just a rubber stamp with better grammar.

The appendix matters because it tests the judge, not the marketing story

The paper’s appendices are useful because they do not merely add decorative detail. They probe whether WebJudge works for reasons that should generalise.

The granularity test compares chain-of-thought outcome judgement with keypoint-wise judgement. Keypoint-wise evaluation performs better when there are three or fewer key points, but worse when there are more. The authors explain the likely reason: generated key points are not always perfectly accurate or necessary. If the evaluator checks every generated key point rigidly, it can become too strict. Combining modes improves overall agreement, but the paper uses the chain-of-thought mode for main experiments because it is cheaper and has a better cost-performance trade-off.

That is an ablation, not a second thesis. Its purpose is to show how evaluation granularity changes reliability and cost.

The threshold test examines how many screenshots WebJudge should keep. A threshold of 3 performs best on Operator trajectories; lower thresholds include too many screenshots and risk overwhelming the model, while higher thresholds discard important evidence. That is a sensitivity test. It supports the mechanism: evaluation quality depends on selecting enough intermediate evidence without flooding the judge.

The robustness test runs WebJudge multiple times and finds low variance, with an average standard deviation of 1.1% across the six agents’ success rates. That matters because LLM evaluators can be non-deterministic even at low temperature. Low variance does not prove perfection, but it reduces one obvious concern: that the metric is too noisy for comparison.

Finally, the AgentRewardBench evaluation tests generalisation across 1,302 trajectories from five other benchmarks. WebJudge variants perform strongly on precision, with o4-mini reaching 82.0% overall precision and WebJudge-7B reaching 75.7%. But the details are important. The o4-mini version is relatively conservative and has lower recall. WebJudge-7B offers a more balanced trade-off and lower cost.

For business readers, this is where the paper becomes more than a benchmark report. It suggests that trajectory-aware evaluators could become reusable infrastructure: for QA, for regression tests, for rejection sampling, for reflection pipelines, and eventually for reinforcement learning. That is useful. It is also not magic. A judge that is conservative may reject acceptable work. A judge that is permissive may approve failures. The balance must match the workflow risk.

For payroll, compliance, and procurement approvals, conservatism is often desirable. For low-risk research browsing, it may be unnecessarily expensive. The judge should fit the court.

The failures are operational, not philosophical

The paper’s error analysis is the most business-relevant section because the failures look familiar.

The authors focus detailed analysis on Operator because it is the strongest evaluated agent. Operator has real strengths: it can use structured search, tool-based navigation such as page search, and self-verification. In one case study, it selects the wrong colour and then corrects itself. That is exactly the sort of behaviour developers want to see.

But the failures are revealing. Operator still struggles with numerical and temporal constraints. In one example, it applies a broader car model-year range than requested. In another, it fails to adjust a flight arrival-time slider correctly. These are not obscure edge cases. They are the bread and butter of business rules.

Other agents show different weaknesses. They may neglect task requirements, hallucinate unmet constraints, explore too little, repeat actions, terminate too early, or rely on loose keyword searches. Browser Use, for example, is shown hallucinating results that do not satisfy location constraints and claiming that jobs are the “most recent” without applying the required sort function.

These are not failures of “AI alignment” in the grand philosophical sense. They are failures of operational discipline.

This is where CognaptusJudge, as a business automation evaluation philosophy, should become stricter than a demo harness. The relevant question is not whether the agent usually gets to a plausible page. The question is whether the audit trail proves that the agent satisfied each business requirement.

For an enterprise workflow, “almost did it” is not a category of success. It is a category of incident report.

What Cognaptus should infer, and what the paper actually shows

It is worth keeping the boundary clean.

The paper directly shows that current web agents perform substantially worse on Online-Mind2Web than some prior WebVoyager-style results suggested. It directly shows that Online-Mind2Web is broader and harder to shortcut than WebVoyager. It directly shows that WebJudge improves agreement with human judgement compared with several existing automatic evaluation methods. It directly shows recurring failure modes around filtering, sorting, navigation, long trajectories, numerical constraints, temporal constraints, and hallucinated final responses.

Cognaptus can reasonably infer that business automation needs similar evaluation discipline. Workflows should be assessed on representative live or near-live tasks. The evaluator should inspect action traces, system states, key screenshots where relevant, and explicit task criteria. Human audit should remain the reference standard for high-risk workflows, while LLM-based judges can reduce the cost of routine evaluation and failure triage.

What remains uncertain is equally important.

Online-Mind2Web is a 300-task benchmark, not the whole internet. It excludes CAPTCHA-protected tasks and constrains Google Search in the main evaluation. Many enterprise workflows happen inside logged-in systems, private databases, ERP platforms, CRM tools, and custom portals that public web benchmarks cannot represent. Some business workflows also have stricter compliance, privacy, and rollback requirements than web browsing tasks.

So the right business conclusion is not to copy Online-Mind2Web wholesale. The right conclusion is to copy its evaluation philosophy.

That means moving from generic success claims to evidence-driven workflow evaluation.

A practical framework for courtroom-style automation QA

A business-ready automation judge should answer five questions.

This structure gives automation teams a practical alternative to inflated demo metrics. Instead of reporting “the agent succeeded 92% of the time” on friendly test cases, they can report success by task class, constraint type, system, exception condition, and evidence quality.

That is less glamorous. It is also how serious operations work.

A CognaptusJudge-style system should therefore be modular. For low-risk workflows, it may use lightweight log evaluation. For visual workflows, it may need selected screenshots. For API-heavy workflows, it should inspect structured state transitions. For regulated workflows, it should preserve audit artefacts and route uncertain cases to humans.

The judge should also be tested against humans. The paper treats human evaluation as the reference standard and measures agreement. Business systems should do the same. An LLM judge that has never been calibrated against human reviewers is not an evaluator. It is another agent wearing a robe.

The ROI is cheaper diagnosis, not blind autonomy

The immediate return from better evaluation is not necessarily full automation. It is cheaper diagnosis.

When an agent fails, the organisation needs to know why. Did it misunderstand the instruction? Did the UI change? Did it apply the wrong filter? Did it fail after a pop-up? Did it hallucinate a completion message? Did the task require a hidden system permission? Each answer leads to a different fix.

Without trajectory-aware evaluation, failures become vague. With it, teams can build a failure taxonomy and improve the workflow systematically.

This matters for three operational reasons.

First, procurement becomes more rational. Vendors can be compared on representative tasks, not polished demos. The evaluation should include easy, medium, and hard workflows, with adversarial cases that reflect real business exceptions.

Second, deployment becomes safer. Agents can be released with confidence thresholds, escalation paths, and audit logs. A system can approve low-risk tasks automatically while routing ambiguous or high-impact cases to humans.

Third, improvement becomes measurable. If failures are categorised, developers can see whether a new model version improves navigation but worsens constraint handling, or whether a prompt update reduces hallucinated completions but increases false failures. That is product management, not vibes management. A radical innovation, apparently.

Boundaries: what this paper does not settle

The paper should not be overread.

It does not prove that web agents cannot be useful today. A 61% success rate on a broad public benchmark can still hide high reliability on a narrow, well-instrumented workflow. Enterprise automation often succeeds by narrowing scope, controlling interfaces, and defining fallbacks.

It does not prove that Google Search should always be blocked. In real work, search may be a legitimate tool. The paper blocks it to measure navigation fairly and avoid shortcut confounds. A production system should decide tool access based on task design, risk, and verification requirements.

It does not prove that WebJudge eliminates human evaluation. Its agreement rates are high enough to be useful, not high enough to replace human audit in high-stakes settings. A judge with 85–87% agreement can accelerate triage and regression testing. It should not be the only control for payments, compliance, medical decisions, legal filings, or anything else where a false pass has serious consequences.

It also does not cover the full complexity of private enterprise systems. Public websites are messy, but they are not the same as a company’s ERP stack, role-based permissions, exception queues, internal naming conventions, or compliance logs.

The boundary is not a weakness. It tells us where to adapt the method. Online-Mind2Web is evidence that evaluation must become more realistic. It is not a universal benchmark for every workflow Cognaptus will ever automate.

The verdict: automation needs evidence before confidence

The paper’s title asks whether progress is partly an illusion. For business automation, the answer is uncomfortable but useful: some of it is.

The illusion comes from benchmarks that are too narrow, tasks that are too shortcut-friendly, and evaluators that believe confident final responses. Once the task setting becomes more realistic and the judge demands trajectory evidence, success rates fall. That is not bad news. It is better measurement.

For Cognaptus, the article’s original courtroom metaphor is exactly right, but it needs sharper teeth. Business automation should not be judged by whether the agent reached a plausible-looking page or produced a persuasive status update. It should be judged by whether the evidence shows that the workflow was completed according to the requirements.

The future of agentic automation will not be won only by better agents. It will also be won by better courts.

And in that courtroom, the agent does not get acquitted because it sounded confident.

Cognaptus: Automate the Present, Incubate the Future.