Model Interpretability

Opening — Why this matters now Large language models appear smooth from the outside: prompts go in, coherent text comes out. But internally, their numerical dynamics are anything but calm. In fact, inside many modern Transformers, certain tokens briefly explode into extreme values thousands of times larger than their neighbors. At the same time, a small set of tokens—often the very first token in a sequence—attracts an overwhelming share of attention from many heads. These are known as attention sinks. ...

Opening — Why this matters now Reasoning models are having a moment. Latent-space architectures promise to outgrow chain-of-thought without leaking tokens or ballooning costs. Benchmarks seem to agree. Some of these systems crack puzzles that leave large language models flat at zero. And yet, something feels off. This paper dissects a flagship example—the Hierarchical Reasoning Model (HRM)—and finds that its strongest results rest on a fragile foundation. The model often succeeds not by steadily reasoning, but by stumbling into the right answer and staying there. When it stumbles into the wrong one, it can stay there too. ...

Imagine debugging a black box. Now imagine that black box occasionally narrates its thoughts aloud. That’s the opportunity—and the fragility—presented by Chain-of-Thought (CoT) monitoring, a newly emergent safety paradigm for large language models (LLMs). In their recent landmark paper, Korbak et al. argue that reasoning traces generated by LLMs—especially those trained for explicit multi-step planning—offer a fleeting yet powerful handle on model alignment. But this visibility, they warn, is contingent, brittle, and already under threat. ...