LLM Monitoring

TL;DR for operators Production monitoring often creates an unattractive choice. A small probe is cheap enough to run on every request but may compress away evidence that occurred earlier in a sequence. A stronger position-aware classifier preserves more information but adds computation to an inference path that is already expensive. Speculative Probing: LLM Monitoring at Speculative-Decoding Cost1 asks whether part of that cost has already been paid. Some LLM deployments use a smaller auxiliary prediction component to accelerate generation. Speculative Probing reuses that component—its speculative-decoding head—as a frozen feature extractor for monitoring. The base model and draft head remain unchanged; only a few learned task-specific query vectors and a small classifier are trained. ...

Audit logs are comforting because they look administrative. A system acts, a trace appears, a reviewer nods, and everyone pretends the record explains the decision. That habit becomes more fragile when the system is an AI model. In many current AI workflows, especially those involving reasoning models or autonomous agents, the chain-of-thought is treated as the closest available thing to an internal audit trail. The model writes down intermediate reasoning, a monitor reads that reasoning, and the organization hopes the dangerous part—deception, hidden goals, sandbagging, sabotage, or simply the decisive cue behind an answer—will be visible before the final action causes trouble. ...

A monitor is only useful if it fails in the boring place. The boring place is production: the real domain, the real prompt style, the real user incentives, the real model generating the real response. Not the tidy benchmark. Not the synthetic dataset. Not the “please pretend to be deceptive” prompt that makes everyone in the lab feel productive. Production is where a detector either catches the thing it was built to catch, or quietly becomes a compliance ornament with a nice AUROC score. ...