TL;DR for operators Mathematical proof is a nasty evaluation setting for AI systems because it leaves fewer hiding places. A model cannot merely land on a final number; it has to preserve the truth of each step. That is precisely why Guo et al.’s RFMDataset is useful: it tests whether advanced reasoning models can construct complete natural-language proofs, then classifies how they fail when they cannot.1 ...
TL;DR for operators The useful lesson is not that vision-language models need longer reasoning traces. They already produce plenty of words. Some of them are even adjacent to thought. The useful lesson is that multimodal systems need feedback that can tell where a reasoning path breaks, not merely whether the final answer looks acceptable. ...
TL;DR for operators Pre-Act is a useful reminder that enterprise agents do not fail only because they choose the wrong tool. They fail because they lose the plot. A customer asks for help, the agent gathers one fact, calls one API, sees an unexpected result, and then behaves as if the workflow has reset. Charming, in the same way a lift that forgets floors is charming. ...
TL;DR for operators A model score is not a certificate. It is a timestamp. That is the operational message of D. Sculley and co-authors’ position paper on GenAI evaluation.1 Their argument is not that every static benchmark is useless, nor that competitions are magical truth machines with leaderboards attached. The argument is sharper: GenAI has broken the old bargain behind machine-learning evaluation. ...
TL;DR for operators Conversational agents are moving from polite text boxes into operational systems: booking, triaging, recommending, retrieving, judging, escalating, and occasionally making a confident mess with impressive formatting. The useful lesson from these two papers is simple: enterprise agents cannot be trusted just because they can reason, remember, or call tools. Those are necessary capabilities, not sufficient safeguards. A serious agent needs a fourth layer: a way to evaluate whether its own decisions and judgments deserve to be used. ...
TL;DR for operators Long-context windows are useful. They are also an expensive way to pretend that memory is just a bigger clipboard. The HEMA paper argues for a more operationally realistic design: keep a compressed summary of the conversation always visible, store detailed past exchanges outside the prompt, and retrieve only the details that matter for the current turn.1 That gives the model two different memory behaviours: continuity from Compact Memory and factual recall from Vector Memory. ...
TL;DR for operators Do not ask, “Can the model do the task?” Ask, “Does the model use the capabilities it already has when the task becomes messy?” Hallucination is not one thing. In a medical, legal, financial, or investment workflow, it is a defect. In a labelled creative mode, it can be a feature. Revolutionary stuff: context matters. Goal-directedness is also not one thing. More goal pursuit can improve execution, but it also raises safety and governance questions. The sensible business pattern is not “deploy an autonomous AI analyst and hope it behaves”. It is mode governance: separate factual, creative, and decision-support modes with different metrics, interfaces, and controls. High-stakes workflows need scaffolding: memory, rule extraction, refinement loops, ensemble checks, scoring, audit trails, and humans who can edit policy rather than merely admire the model’s prose. AI products are currently being sold with a suspiciously convenient promise: one conversational interface will reason, search, write, create, decide, advise, analyse, and maybe spiritually support the quarterly planning meeting if procurement approves the invoice. ...
TL;DR for operators The AI market is not choosing between “one model to rule them all” and “a thousand specialist flowers blooming politely in a procurement spreadsheet.” It is choosing by workload. GPT-4o’s native image generation matters because it folds visual production into the same conversational workspace where users already brainstorm, rewrite, code, and revise. That is not just a model upgrade. It is a distribution upgrade. The GPT-4o system card describes an omni model trained across text, vision, and audio, with stronger multimodal capability and lower API cost than GPT-4 Turbo in OpenAI’s own framing.1 OpenAI’s March 2025 image-generation release then pushed that logic into visual work: generate, critique, revise, and regenerate without leaving the chat.2 ...
TL;DR for operators CoRAG is not “RAG, but with more documents.” It is a way to let multiple organizations train a shared retrieval-augmented model while keeping their labeled question-answer data local. That matters because labels are usually the expensive, sensitive, commercially revealing part. Market documents, manuals, policies, public reports, and technical references are often easier to share than the annotations that say which answer was correct, for whom, and under what business condition. Tiny distinction. Large legal bill avoided. ...
TL;DR for operators Enterprise agents fail less like philosophers and more like junior coordinators with access to the wrong dropdown menu. They propose actions that are not currently possible. They miss actions that are possible. They forget that an action changes the world. They treat impossible future states as if determination will somehow make them available. They add redundant steps, skip mandatory subgoals, or pick a next move that feels plausible but does not reduce the distance to the goal. ...