RAG

Exams are useful because they are rude. They do not care that a model sounds polished, cites the right buzzwords, or can produce a gorgeous paragraph about duration risk. They ask for A, B, or C. Then they mark the answer wrong. That is why a new CFA-based benchmark is more useful than another misty-eyed essay about AI “transforming finance.” The paper evaluates GPT-4o, GPT-o1, and o3-mini on 1,560 official CFA mock multiple-choice questions across Levels I, II, and III, both zero-shot and with a domain-reasoning RAG pipeline built from official CFA curriculum materials.1 The result is not a single leaderboard. It is closer to a routing manual. ...

A RAG system usually fails in one of two annoyingly familiar ways. It retrieves documents that are factually relevant but gives the model no clue about the task’s decision boundary. Or it retrieves labelled examples that show the decision pattern but are too parochial to help when the topic drifts. One source knows the world. The other knows the exam rubric. Naturally, many systems pick one and then pretend the compromise was strategy. ...

TL;DR for operators Legal AI does not fail only because models “hallucinate”. That word has become the industry’s favourite fog machine. The more operational diagnosis is sharper: models fail when they answer current legal questions from stale internal memory and then dress the error in confident reasoning. The L-MARS paper is useful because it separates two tasks that vendors often blend together for convenience: retrieving current legal facts and reasoning over stable legal principles.1 On LegalSearchQA, a new 50-question benchmark built around recent U.S. legal facts verified in March 2026, L-MARS reaches 96.0% accuracy. Zero-shot GPT-4o-mini reaches 58.0%. Chain-of-thought falls to 30.0%, because step-by-step reasoning from outdated premises merely creates a more articulate mistake. ...

TL;DR for operators Many AI workflows do not need a yes-or-no judgment. They need a number: how well did this answer follow the instruction, how far did this reasoning trace remain valid, how much better is answer A than answer B, how strong is this essay, how risky is this case, how close is this support call to escalation? ...

TL;DR for operators LLM application testing should stop pretending that the whole product behaves like ordinary software. The database connector, retry logic, API wrapper, and schema validator still deserve normal unit, integration, and load tests. Fine. Keep those. They are not the problem. The problem starts when the product becomes a stateful language system: prompts are assembled dynamically, retrieval changes the context, tool calls modify the execution path, memory leaks across turns, and a model update can improve one workflow while quietly breaking another. At that point, exact-match assertions become less like QA and more like theatre with a YAML file. ...

TL;DR for operators GLARE is useful because it attacks the boring but expensive failure mode in legal AI: the model jumps to the familiar label, decorates the guess with legal-sounding prose, and hopes nobody asks whether a nearby charge would have fit better. The paper proposes an agentic legal judgment prediction framework that does three things in sequence: it expands the set of candidate charges, retrieves precedents with explicit reasoning paths rather than just similar facts, and performs targeted legal search when the model detects a knowledge gap.1 That mechanism matters more than the branding. GLARE is not “RAG, but with legal documents.” It is closer to a small operating procedure for legal reasoning: widen the hypothesis space, compare alternatives, then fetch the missing premise. ...

TL;DR for operators Most enterprise LLM failures do not come from the model “not knowing enough”. They come from the system forgetting what it was doing five minutes ago, rediscovering the same facts, and treating every user turn as a fresh episode in a soap opera nobody asked to watch. The paper behind this article proposes Cognitive Workspace: an active memory architecture for LLMs that deliberately curates, reuses, consolidates, and forgets information rather than merely retrieving chunks or stretching the context window.1 Its core claim is simple but consequential: useful long-context behaviour is not the same as having a long context window. It is the ability to maintain a working state across a task. ...

TL;DR for operators Research agents fail in a very familiar way: they do several useful things, then make one bad final move, and the training signal treats the whole journey as garbage. Delightful. Efficient. Totally not a credit-assignment problem wearing a lab coat. Atom-Searcher attacks that problem by splitting an agent’s reasoning trace into Atomic Thoughts: small, functional reasoning units such as planning, verification, hypothesis testing, observation, action selection, or risk analysis. A Reasoning Reward Model then scores those units, producing an Atomic Thought Reward that is blended with the final-answer reward during reinforcement learning.1 ...

TL;DR for operators CryptoScope is not “ChatGPT, please audit my cryptography”. That would be a splendid way to generate confident nonsense with Greek letters. The paper’s useful idea is more disciplined: make the model behave less like a wandering code reviewer and more like a junior cryptographic analyst with a library card, a checklist, and a supervisor. CryptoScope does this by combining three components: a curated cryptographic knowledge base of more than 12,000 entries, a pre-detection step that summarises code and checks algorithm compliance, and a retrieval-augmented final analysis that grounds the model’s reasoning in known failure patterns and implementation guidance.1 ...

TL;DR for operators Compliance teams do not mainly need a chatbot that sounds more confident. They already have enough people sounding confident in meetings. They need answers that can be traced back to the rule text, checked against related provisions, and updated when the regulatory corpus changes. The paper behind this article proposes a multi-agent system that turns regulatory documents into subject–predicate–object triplets, embeds those triplets alongside their source sections, retrieves triplets for question answering, and shows users the relevant subgraph behind the answer.1 That matters because regulatory work is not just “find me a paragraph.” It is “show me the applicable rule, the linked requirement, the exception, the deadline, and the neighbouring clause that will embarrass us later.” ...