Evaluation

A customer tells your AI assistant that she dislikes crowded tourist attractions. Three weeks later, she asks for a weekend itinerary. A good assistant should not proudly recommend the busiest landmark in the city. A less good assistant will do exactly that, but in a warm tone. This is the quiet failure mode behind many “personal AI” demos. The interface remembers the conversation. The product claims continuity. The model may even have a giant context window large enough to swallow a small novel. Yet when the user asks a new question, the system behaves as if the earlier preference is just decorative text floating somewhere in the attic. ...

Benchmarks are the places where AI systems go to look impressive. That is not automatically a problem. A good benchmark clarifies what a system can do, what it cannot do, and where progress is real. A bad benchmark performs a more theatrical function: it lets researchers win a carefully chosen game, write a confident conclusion, and quietly hope nobody asks whether the result survives contact with another task. ...

Checklist. It is not the most glamorous word in artificial intelligence. It does not sound like a new reasoning architecture, a sovereign model, or a mildly terrifying demo video. It sounds like something an operations manager would use before approving a vendor payment. That is exactly why it matters. Most enterprise agents fail to fit the clean reward structure that reinforcement learning likes. A coding benchmark can verify whether tests pass. A math problem can verify the final answer. A database query can sometimes verify whether a returned value matches the expected record. But business agents live in a less cooperative universe. They ask clarification questions, call internal tools, respect constraints, recover from missing information, and produce replies that are useful without being exactly predictable. ...

A robot does not fail politely. It does not say, “I was trained on a slightly different shade of blue.” It just misses the object, pushes the wrong way, or confidently follows a plan that only works in the tidy little universe where the benchmark was born. That is the uncomfortable lesson behind stable-worldmodel-v1, a paper that is less about inventing a new world model and more about asking whether world-model research has been measuring the right thing in the first place.1 ...

Screenshots are easy to love. They sit still, look polished, and ask very little from the viewer. Interfaces are less polite. Click one wrong icon, place a menu twenty pixels away from where it belongs, blur one label, or forget what happened three screens ago, and the whole interaction becomes decorative theatre. ...

The room is not impressed by your leaderboard A robot that performs well on a public benchmark has not necessarily learned how to operate in your house. It may recognize a chair in a dataset. It may answer a visual question about a tidy image. It may even produce a confident paragraph explaining where the coffee mug should be. Then it enters a real room — with mirrors, partial views, cluttered corners, awkward sightlines, and objects that are not positioned for benchmark convenience — and suddenly the “general intelligence” starts behaving like a tourist holding the map upside down. ...

A demo can make an AI research agent look impressive in ten minutes. Give it a task, watch it create files, install packages, run experiments, generate tables, and write something that sounds like a conclusion. Productivity theater, now with terminal logs. The harder question is less cinematic: did it actually discover the right thing? ...

A model can generate a pretty sequence of images. Good. So can a slide deck. The harder question is whether those images actually help it think. That is the uncomfortable point behind MentisOculi: Revealing the Limits of Reasoning with Mental Imagery, a new benchmark paper that tests whether frontier multimodal models can do something closer to human mental imagery: form a visual state, keep it stable, transform it step by step, and use the transformed state to decide what to do next.1 Not merely “look at an image and answer a question.” Not “draw a plausible intermediate picture.” Actual visual reasoning, with consequences. ...

Dashboard numbers are seductive because they look obedient. Revenue goes up, traffic dips, latency spikes, inventory turns over, temperature drifts, volatility clusters. Put the sequence into a chart and the pattern seems almost polite. Then someone asks an LLM what happened. The model answers fluently. It may even sound like an analyst who has seen too many quarterly review decks and has developed a protective layer of confidence. But fluency is not temporal understanding. A model can describe a curve, name a trend, and still fail to understand which segment comes next, whether a transformation is correct, or whether a discontinuity is an error or a legitimate feature of the process. ...

A chatbot can act like a pirate for three turns. That is not the impressive part. A teenager with a Halloween hat can also do that. The harder problem begins when the agent has to remember what happened last week, preserve a recognizable personality across changing situations, make choices consistent with its motives, avoid borrowing another character’s copyrighted voice a little too enthusiastically, and still behave safely when the user pushes it outside the script. At that point, “pretend you are X” stops being a prompt trick and becomes a systems engineering problem. ...