LLM-as-Judge

The procurement meeting is where public leaderboards go to look useful Benchmark scores are comforting because they compress chaos into a number. One model is 87.3, another is 84.9, and suddenly the procurement meeting has the emotional texture of financial discipline. Very mature. Very measurable. Also, very possibly irrelevant. The problem is simple. A company rarely wants “the best model on average”. It wants the best model for contract review, support triage, clinical note summarisation, SQL repair, claims handling, product search, or whatever unglamorous workflow actually pays the cloud bill. Public benchmarks are often too generic for that decision. Worse, the benchmark items may already be floating inside model training data, turning evaluation into a memory test with better typography. ...

Portfolio committees have a talent for making slow processes look dignified. The ritual is familiar: an Investment Policy Statement sets the mandate, analysts prepare capital market assumptions, consultants run an optimizer or two, the investment committee meets, the board receives a memo, and everyone hopes the assumptions survive until the next review cycle. It is not irrational. It is simply bounded by human attention, calendar slots, model-maintenance capacity, and the fact that even very clever people cannot run twenty competing allocation philosophies before lunch. ...

A support agent can sound calm, ask polite questions, invoke a few tools, and finish with a reassuring summary. The customer leaves. The dashboard shows completion. Everyone feels civilized. Then someone opens the actual transaction log. The reservation was not cancelled. The reminder was searched before the timestamp was retrieved. The contact update succeeded for the wrong person. The model was not exactly malicious, or even spectacularly wrong. It was simply agreeable in the familiar corporate way: fluent enough to pass the meeting, not reliable enough to run the process. ...

The evaluator is not the scale Evaluation looks boring until it changes the winner. A product team compares three candidate responses. A benchmark ranks five model releases. A content workflow asks an LLM judge to score generated SEO packs. The spreadsheet fills itself politely: five rubric dimensions, an overall score, maybe a few quoted receipts. Everyone pretends the judge is just a thermometer. ...

A hospital does not need a perfect theory of neural network internals before it can notice that one clinical AI keeps recommending the wrong kind of follow-up. A bank does not need to decode every transformer layer before it can see that a credit assistant behaves oddly around post-bankruptcy applicants. A regulator does not need metaphysics. It needs repeatable measurements. ...

A tutoring system has one deceptively simple job: give the learner the next problem. Not the hardest problem. Not the flashiest problem. Not the one that makes the model feel terribly pleased with itself after a 4,000-token monologue. The next problem: the one that keeps a student engaged, teaches the right structure, and feels worth the effort. ...

TL;DR for operators Conversational agents are moving from polite text boxes into operational systems: booking, triaging, recommending, retrieving, judging, escalating, and occasionally making a confident mess with impressive formatting. The useful lesson from these two papers is simple: enterprise agents cannot be trusted just because they can reason, remember, or call tools. Those are necessary capabilities, not sufficient safeguards. A serious agent needs a fourth layer: a way to evaluate whether its own decisions and judgments deserve to be used. ...

TL;DR for operators A web agent that looks impressive in a demo may still fail when asked to complete ordinary live tasks across messy websites. That is the central finding of An Illusion of Progress? Assessing the Current State of Web Agents, which introduces Online-Mind2Web, a benchmark of 300 realistic tasks across 136 websites.1 ...