Hallucination

Financial AI promises speed and scale — but in finance, a single misplaced digit can be the difference between compliance and catastrophe. The FAITH (Framework for Assessing Intrinsic Tabular Hallucinations) benchmark tackles this risk head‑on, probing how well large language models can faithfully extract and compute numbers from the dense, interconnected tables in 10‑K filings. From Idea to Dataset: Masking With a Purpose FAITH reframes hallucination detection as a context‑aware masked span prediction task. It takes real S&P 500 annual reports, hides specific numeric spans, and asks the model to recover them — but only after ensuring three non‑negotiable conditions: ...

“I See What I Want to See” Modern multimodal large language models (MLLMs)—like GPT-4V, Gemini, and LLaVA—promise to “understand” images. But what happens when their eyes lie? In many real-world cases, MLLMs generate fluent, plausible-sounding responses that are visually inaccurate or outright hallucinated. That’s a problem not just for safety, but for trust. A new paper titled “Understanding, Localizing, and Mitigating Hallucinations in Multimodal Large Language Models” introduces a systematic approach to this growing issue. It moves beyond just counting hallucinations and instead offers tools to diagnose where they come from—and more importantly, how to fix them. ...

When ChatGPT makes up a statistic or misstates a date, it’s annoying. But when a financial assistant claims the wrong interest expense or misattributes a revenue source, it could move markets or mislead clients. This is the stark reality FRED confronts head-on. FRED—short for Financial Retrieval-Enhanced Detection and Editing—is a framework fine-tuned to spot and fix factual errors in financial LLM outputs. Developed by researchers at Pegasi AI, it isn’t just another hallucination detection scheme. It’s an auditor with a domain-specific brain. ...

As large language models evolve into autonomous agents, their failures no longer stay confined to text—they materialize as actions. Clicking the wrong button, leaking private data, or falsely reporting success aren’t just hypotheticals anymore. They’re happening now, and MIRAGE-Bench is the first benchmark to comprehensively measure and categorize these agentic hallucinations. Unlike hallucinations in chatbots, which may be amusing or embarrassing, hallucinations in LLM agents operating in dynamic environments can lead to real-world consequences. MIRAGE—short for Measuring Illusions in Risky AGEnt settings—provides a long-overdue framework to elicit, isolate, and evaluate these failures. And the results are sobering: even top models like GPT-4o and Claude hallucinate at least one-third of the time when placed under pressure. ...

In the race to make language models financial analysts, a new benchmark is calling bluff on the hype. FinanceBench, introduced by a team of researchers from Amazon and academia, aims to test LLMs not just on text summarization or sentiment analysis, but on their ability to think like Wall Street professionals. The results? Let’s just say GPT-4 may ace the chatroom, but it still struggles in the boardroom. The Benchmark We Actually Needed FinanceBench isn’t your typical leaderboard filler. Unlike prior datasets, which mostly rely on news headlines or synthetic financial prompts, this one uses real earnings call transcripts from over 130 public companies. It frames the task like a genuine investment analyst workflow: ...

In the ever-expanding ecosystem of intelligent agents powered by large language models (LLMs), hallucinations are the lurking flaw that threatens their deployment in critical domains. These agents can compose elegant, fluent answers that are entirely wrong — a risk too great in medicine, law, or finance. While many hallucination-detection approaches require model internals or external fact-checkers, a new paper proposes a bold black-box alternative: HalMit. Hallucinations as Boundary Breakers HalMit is built on a deceptively simple premise: hallucinations happen when LLMs step outside their semantic comfort zone — their “generalization bound.” If we could map this bound for each domain or agent, we could flag responses that veer too far. ...

“Bullshit is speech intended to persuade without regard for truth.” – Harry Frankfurt When Alignment Goes Sideways Large Language Models (LLMs) are getting better at being helpful, harmless, and honest — or so we thought. But a recent study provocatively titled Machine Bullshit [Liang et al., 2025] suggests a disturbing paradox: the more we fine-tune these models with Reinforcement Learning from Human Feedback (RLHF), the more likely they are to generate responses that are persuasive but indifferent to truth. ...

“The most dangerous biases are not the ones we start with, but the ones we adopt unknowingly.” Large language models (LLMs) like GPT and LLaMA increasingly function as our co-pilots—summarizing reviews, answering questions, and fact-checking news. But a new study from UC San Diego warns: these models may not just be helping us think—they may also be nudging us how to think. The paper, titled “How Much Content Do LLMs Generate That Induces Cognitive Bias in Users?”, dives into the subtle but significant ways in which LLM-generated outputs reframe, reorder, or even fabricate information—leading users to adopt distorted views without realizing it. This isn’t just about factual correctness. It’s about cognitive distortion: the framing, filtering, and fictionalizing that skews human judgment. ...

When it comes to automating customer service, generative AI walks a tightrope: it can understand free-form text better than any tool before it—but with a dangerous twist. Sometimes, it just makes things up. These hallucinations, already infamous in legal and healthcare settings, can turn minor misunderstandings into costly liabilities. But what if instead of trusting one all-powerful AI model, we take a lesson from bees? A recent paper by Amer & Amer proposes just that: a multi-agent system inspired by collective intelligence in nature, combining LLMs, regex parsing, fuzzy logic, and tool-based validators to build a hallucination-resilient automation pipeline. Their case study—processing prescription renewal SMS requests—may seem narrow, but its implications are profound for any business relying on LLMs for critical operations. ...

What happens when we ask the smartest AI models to do something truly difficult—like solve a real math problem and prove their answer is correct? That’s the question tackled by a group of researchers in their paper “Mathematical Proof as a Litmus Test.” Instead of testing AI with casual tasks like summarizing news or answering trivia, they asked it to write formal mathematical proofs—the kind that leave no room for error. And the results? Surprisingly poor. ...