Model Evaluation

Opening — Why this matters now The industry loves to talk about generalization. We celebrate models that extrapolate, reason, and improvise. But lurking underneath this narrative is a less glamorous behavior: memorization. Not the benign kind that helps recall arithmetic, but the silent absorption of training data—verbatim, brittle, and sometimes legally radioactive. The paper behind this article asks a pointed question the AI industry has mostly tiptoed around: where, exactly, does memorization happen inside large language models—and how can we isolate it from genuine learning? ...

Opening — Why this matters now For years, the dominant anxiety around large language models has been hallucination: the model makes things up. The paper you just read argues that we’ve been staring at the wrong failure mode. The real issue is subtler and arguably more dangerous: memorization sinks — regions of the training distribution where models stop learning general structure and instead collapse into rote recall. These sinks don’t merely inflate benchmark scores; they quietly reshape model behavior, evaluation outcomes, and downstream reliability. ...

Opening — Why this matters now Large language models are getting better at everything—writing, coding, reasoning, and politely apologizing when they hallucinate. Yet beneath these broad performance gains lies a quieter, more structural issue: memorization does not happen evenly. Some parts of the training data exert disproportionate influence, acting as gravitational wells that trap model capacity. These are what the paper terms memorization sinks. ...

Bias on Demand: When Synthetic Data Exposes the Moral Logic of AI Fairness In the field of machine learning, fairness is often treated as a technical constraint — a line of code to be added, a metric to be optimized. But behind every fairness metric lies a moral stance: what should be equalized, for whom, and at what cost? The paper “Bias on Demand: A Modelling Framework that Generates Synthetic Data with Bias” (Baumann et al., FAccT 2023) breaks this technical illusion by offering a framework that can manufacture bias in data — deliberately, transparently, and with philosophical intent. ...

TL;DR Most demos of “LLM introspection” are actually vibe checks on outputs, not privileged access to internal state. If a third party with the same budget can do as well as the model “looking inward,” that’s not introspection—it’s ordinary evaluation. Two quick experiments show temperature self‑reports flip with trivial prompt changes and offer no edge over across‑model prediction. The bar for introspection should be higher, and business users should demand it. ...

When Pixar’s Inside Out dramatized the mind as a control room of core emotions, it didn’t imagine that language models might soon build a similar architecture—on their own. But that’s exactly what a provocative new study suggests: large language models (LLMs), without explicit supervision, develop hierarchical structures of emotions that mirror human psychological models like Shaver’s emotion wheel. And the larger the model, the more nuanced its emotional understanding becomes. ...

In the age of Transformers and neural nets that write poetry, it’s tempting to assume deep learning dominates every corner of AI. But in quantitative investing, the roots tell a different story. A recent paper—QuantBench: Benchmarking AI Methods for Quantitative Investment1—delivers a grounded reminder: tree-based models still outperform deep learning (DL) methods across key financial prediction tasks. ...