LLM Agents

Most LLM agents today think in flat space. When you ask a long-term assistant a question, it either scrolls endlessly through past turns or scours an undifferentiated soup of semantic vectors to recall something relevant. This works—for now. But as tasks get longer, more nuanced, and more personal, this memory model crumbles under its own weight. A new paper proposes an elegant solution: H-MEM, or Hierarchical Memory. Instead of treating memory as one big pile of stuff, H-MEM organizes past knowledge into four semantically structured layers: Domain, Category, Memory Trace, and Episode. It’s the difference between a junk drawer and a filing cabinet. ...

The dominant paradigm in LLM agents today is autoregressive reasoning: think step by step, commit token by token. This approach works decently for small tasks — write a tweet, answer a math question — but it quickly falters when the goal requires deep planning, multiple decision branches, or adapting to partially observable environments. Imagine trying to plan a vacation or operate a flight search website while thinking only one move ahead. ...

If today’s AI agents are so good with tools, why are they still so bad with people? That’s the uncomfortable question posed by UserBench, a new gym-style benchmark from Salesforce AI Research that evaluates LLM-based agents not just on what they do, but how well they collaborate with a user who doesn’t say exactly what they want. At first glance, UserBench looks like yet another travel planning simulator. But dig deeper, and you’ll see it flips the standard script of agent evaluation. Instead of testing models on fully specified tasks, it mimics real conversations: the user’s goals are vague, revealed incrementally, and often expressed indirectly. Think “I’m traveling for business, so I hope to have enough time to prepare” instead of “I want a direct flight.” The agent’s job is to ask, interpret, and decide—with no hand-holding. ...

As large language models evolve into autonomous agents, their failures no longer stay confined to text—they materialize as actions. Clicking the wrong button, leaking private data, or falsely reporting success aren’t just hypotheticals anymore. They’re happening now, and MIRAGE-Bench is the first benchmark to comprehensively measure and categorize these agentic hallucinations. Unlike hallucinations in chatbots, which may be amusing or embarrassing, hallucinations in LLM agents operating in dynamic environments can lead to real-world consequences. MIRAGE—short for Measuring Illusions in Risky AGEnt settings—provides a long-overdue framework to elicit, isolate, and evaluate these failures. And the results are sobering: even top models like GPT-4o and Claude hallucinate at least one-third of the time when placed under pressure. ...

In early 2025, Apple’s now-infamous “thinking-illusion” benchmark delivered a sobering verdict: large reasoning models (LRMs)—those step-by-step thinkers like DeepSeek-R1 and Qwen 3 Thinking—failed to show meaningful advantages over simpler LLMs. Their verbose, reflective outputs didn’t help on easy problems, nor did they scale on hard ones. In some cases, they even underperformed. But what if we were judging thinking models under unfair conditions? A new study titled “Thinking Isn’t an Illusion” argues that the problem isn’t with reasoning itself—it’s with reasoning in a vacuum. When these models are augmented with tools like Python interpreters and structured scratchpads, their performance transforms dramatically. In fact, they begin to consistently outperform their non-reasoning counterparts across a diverse set of logic puzzles. ...

In the ever-expanding ecosystem of intelligent agents powered by large language models (LLMs), hallucinations are the lurking flaw that threatens their deployment in critical domains. These agents can compose elegant, fluent answers that are entirely wrong — a risk too great in medicine, law, or finance. While many hallucination-detection approaches require model internals or external fact-checkers, a new paper proposes a bold black-box alternative: HalMit. Hallucinations as Boundary Breakers HalMit is built on a deceptively simple premise: hallucinations happen when LLMs step outside their semantic comfort zone — their “generalization bound.” If we could map this bound for each domain or agent, we could flag responses that veer too far. ...

As LLM-powered agents become the backbone of many automation systems, their ability to reliably invoke external tools is now under the spotlight. Despite impressive multi-step reasoning, many such agents crumble in practice—not because they can’t plan, but because they can’t parse. One wrong parameter, one mismatched data type, and the whole chain collapses. A new paper titled “Butterfly Effects in Toolchains” offers the first systematic taxonomy of these failures, exposing how parameter-filling errors propagate through tool-invoking agents. The findings aren’t just technical quirks—they speak to deep flaws in how current LLM systems are evaluated, built, and safeguarded. ...

The promise of fully autonomous vehicles hinges on their ability to handle not just the average drive—but the unexpected. Yet, creating rare, safety-critical scenarios for testing autonomous driving (AD) systems has long been a bottleneck. Manual scene creation doesn’t scale. Generative models often drift away from real-world distributions. And collecting edge cases on the road? Too dangerous, too slow. Enter AGENTS-LLM, a deceptively simple yet powerful framework that uses Large Language Models (LLMs) not to solve traffic scenes, but to break them. The twist? These aren’t just static prompts or synthetic scripts. AGENTS-LLM organizes LLMs into a multi-agent, modular system that modifies real traffic scenarios with surgical precision—making them trickier, nastier, and far more useful for evaluating planning systems. ...

If you’ve ever tried to simulate user behavior using LLMs, you’ve probably noticed the same frustrating pattern: the agents are too polite, too helpful, and too similar. They lack the kind of quirks, inconsistencies, and contextually grounded views that make real people interesting—and unpredictable. Enter TinyTroupe, Microsoft’s new open-source toolkit that flips the script on LLM-agent design. Instead of building yet another task-oriented assistant or collaborative workflow bot, TinyTroupe takes the form of a behavioral simulation laboratory. It invites us to think of agents not as obedient coworkers, but as idiosyncratic personas—each with their own backstories, beliefs, and sometimes maddening biases. ...

In the race to build autonomous software engineers, large language model (LLM) agents like Devin and Copilot Chat are lauded for fixing bugs, writing code, and even completing tasks from GitHub issues. But what happens when the code doesn’t even run? That’s the uncomfortable gap SetupBench aims to measure—and the results are sobering. SetupBench introduces a 93-task benchmark evaluating a foundational but under-tested skill: bootstrapping a development environment from scratch. Unlike prior benchmarks that hand agents a fully pre-configured Docker container, SetupBench drops them into a barebones Linux sandbox and challenges them to install dependencies, initialize databases, configure background services, and resolve real-world version conflicts. It sounds simple. It isn’t. ...