AI Evaluation

Search looks easy until the answer is hiding in a caption, a cropped image region, a second web page, and one annoyingly necessary intermediate clue. That is the problem BrowseComp-V3 is trying to measure.1 Not whether a multimodal model can recognize an object in an image. Not whether a chatbot can summarize the first search result. Not whether a web agent can click around long enough to look busy. The benchmark asks a more operationally relevant question: can an AI system browse the open web, combine text and visual evidence across multiple steps, and still arrive at the right answer? ...

A support ticket arrives. The agent reads the same customer history, sees the same policy document, and has access to the same tools. On Monday, it searches for the refund rule, retrieves the correct clause, and gives a clean answer. On Tuesday, with the same input, it searches for a different phrase, retrieves a less relevant document, wanders through two extra steps, and ends with a confident answer that is only approximately useful. ...

Geometry looks clean. A cube has edges. A projection has rules. A missing view should follow from the views already shown. This is not the messy world of occluded street scenes, motion blur, shadows, or a warehouse camera pointed at the wrong shelf. It is the kind of visual reasoning many students learn before they are trusted with anything more dangerous than a compass, a ruler, and mild boredom. ...

Taxi dispatch is not a poetry recital. When a passenger calls and says, “I’m on Arguello,” the system does not need to appreciate the full expressive richness of the sentence. It needs to identify one street name, map it to the right place, and send a vehicle there. This is not a broad language-understanding task. It is a narrow operational task with coordinates attached. ...

Recipe localization looks like an easy prompt. “Create a Jamaican version of Moroccan couscous.” The model smiles politely, throws in jerk seasoning, allspice, scotch bonnet, maybe coconut milk if it is feeling ambitious, and returns something that looks country-specific enough to survive a quick marketing review. The title says “Jamaican.” The ingredients sound Jamaican. The format is clean. No hallucinated oven temperature from another dimension. Excellent, ship it. ...

A customer-service agent rebooks a flight, checks a policy, calls an API, updates the passenger record, apologizes politely, and still gets the outcome wrong. The old explainability question would be: which input tokens influenced the final answer? That question is not useless. It is just late to the crime scene. When an AI system only predicts, explanation can focus on a single input-output decision. When an AI system acts, explanation has to follow the behavior across time: the state it maintained, the tool it selected, the observations it received, the recovery move it attempted, and the point where the run quietly became unrecoverable. A nice feature-importance chart does not tell you that. It tells you what mattered to a prediction, not how a workflow failed. ...

Proof is where AI systems stop performing confidence and start owing the reader money. A model can restate a theorem elegantly. It can cite the right neighborhood of literature. It can produce LaTeX with the visual manners of a publishable paper. None of that is a proof. It is proof-shaped material. Sometimes useful. Sometimes impressive. Sometimes a very expensive fog machine. ...

A leaderboard is a comforting object. It gives procurement teams, product managers, and slightly sleep-deprived founders the same small pleasure: a ranked list. Bigger number, better model. Lower rank, worse model. Decision made. Spreadsheet closed. Everyone can return to pretending vendor evaluation is objective. Unfortunately, benchmarks do not care what your business actually needs. ...

The model is not always confused. Sometimes it has already done the work, reached the right answer, and then politely walks away from it because the user sounded confident. That is the quietly irritating problem behind RAudit, a paper that studies how large language models behave when their reasoning is audited without giving the auditor the correct answer.1 The paper is not just another “LLMs can be sycophantic” warning. We have enough of those. At this point, saying models flatter users is like saying spreadsheets contain hidden errors. True, useful, and somehow still not enough to change deployment practice. ...

Image generation has become good enough to be useful and unreliable enough to remain annoying. That is the normal condition of enterprise AI: impressive demos, awkward edge cases, and someone in operations quietly asking whether the model actually understood the instruction or merely produced something that looked plausible from a distance. A user asks for “a red ceramic mug on a wooden desk, next to an open notebook, in morning light.” The model produces a beautiful desk, credible sunlight, maybe even the notebook. The mug is blue. Or metallic. Or missing. If a separate vision model can look at the image and say, “That is not a red ceramic mug,” the failure feels almost rude. The system can see the problem after creating it. Very efficient, in the same way that a committee can discover a typo after approving the brochure. ...