The Guard Is Already in the Draft: Reusing Speculative Decoding for LLM Monitoring

TL;DR for operators

Production monitoring often creates an unattractive choice. A small probe is cheap enough to run on every request but may compress away evidence that occurred earlier in a sequence. A stronger position-aware classifier preserves more information but adds computation to an inference path that is already expensive.

Speculative Probing: LLM Monitoring at Speculative-Decoding Cost¹ asks whether part of that cost has already been paid. Some LLM deployments use a smaller auxiliary prediction component to accelerate generation. Speculative Probing reuses that component—its speculative-decoding head—as a frozen feature extractor for monitoring. The base model and draft head remain unchanged; only a few learned task-specific query vectors and a small classifier are trained.

The production case is conditional. If the speculative head is already running and its cached sequence representations are already resident in memory, monitoring can reuse them rather than run another large classifier. But the method is neither generally free nor supervision-free: the reported results depend on an existing, sufficiently capable draft head and thousands of labeled examples for most tasks. The paper also provides stronger evidence about asymptotic added computation than about end-to-end wall-clock latency in varied production systems.

Cheap monitors can lose evidence inside the sequence

Suppose a production team wants to flag sensitive information, instruction conflicts, repetitive reasoning, or unsafe prompts while a model is running. The least expensive approach is to take one pooled representation—often the final token state—and train a small classifier on top of it.

That compression can matter.

In the paper’s controlled PII experiment, inputs were truncated to 2,048 tokens and sensitive spans were injected into long documents. A two-layer last-token MLP probe reached 72% accuracy. Two methods able to access position-specific information, MultiMax and Speculative Probing with one learned query step, both reached 100%.

This experiment is a controlled stress test, not evidence that last-token probes will fail by 28 percentage points on ordinary production traffic. Its role is more specific: it demonstrates a failure mode in which relevant evidence exists somewhere in a long sequence but is not reliably preserved in one pooled state.

The next question is whether recovering that sequence information requires another substantial inference path.

The monitoring query runs through computation already used for decoding

Speculative decoding accelerates generation by using a smaller auxiliary component to propose future tokens. That auxiliary component has already processed the sequence and stored reusable internal representations of the prefix. Those stored keys and values form its KV cache.

Speculative Probing leaves both the main LLM and this draft head frozen. Instead of asking the draft head only for token predictions, it appends a small learned task-specific vector that can attend over the cached sequence. The paper calls this a soft prompt.

For SP-1, there is one such query. SP-2 and SP-5 recursively feed two or five learned query positions through the draft head. The final hidden representation $\mathbf{a}$ is then reduced to a binary decision with a linear classifier:

Only the soft prompts and $\mathbf{w}$ are trained.

The computational argument comes from reuse. The draft head has already generated and retained the prefix cache, so the monitor only needs to compute attention and feed-forward operations for the new query positions. The paper characterizes the additional work for $k$ prompt positions as

rather than recomputing the full $n$-token prefix through another model.

The monitoring states are also described as scratch states: they run off the decoding path rather than becoming part of the persistent drafter cache. The proposed monitor therefore reads from speculative-decoding infrastructure without intentionally changing the drafter’s normal token-generation state.

The benchmarks support an efficiency trade-off, not universal dominance

The main evaluation covers four binary monitoring tasks across Qwen3.5-4B, 9B, and 27B models with native MTP heads, plus MiniCPM4.1-8B with an Eagle3 head.

The pattern is fairly consistent: sequence-aware probes matter most when the task depends on structure distributed across the input.

On VerIH instruction-contradiction detection, for example, the Qwen3.5-4B last-token MLP reached 77.7%, while MultiMax reached 91.3% and SP-5 reached 92.3%. On Qwen3.5-27B reasoning-strategy classification, the MLP reached 74.5%, MultiMax 81.5%, and SP-1 85.0%.

The advantage is smaller for detecting repetitive reasoning. There the MLP is already strong: on Qwen3.5-9B it reached 89.5%, versus 91.5% for SP-1.

Across the paper’s 16 model-task cells, Speculative Probing matches or exceeds MultiMax in half. In most of the remaining cells it trails by roughly two to four percentage points. That makes the comparison primarily an accuracy-versus-added-computation result rather than a claim that Speculative Probing is always the most accurate classifier.

The multilingual safety results illustrate the same boundary. Qwen3.5-27B with SP-5 reached 78.2%, compared with 78.1% for the reported zero-shot GPT-5.4-mini reference, 74.5% for Qwen3Guard-Gen-8B, and 67.3% for Llama-Guard-3-8B. Those numbers show competitiveness on this benchmark; they do not establish replacement of dedicated guard models across safety distributions.

One transfer test changes how the in-domain leaderboard should be read

The paper adds one explicit cross-corpus safety test. A Qwen3.5-9B probe trained on Nemotron Safety was evaluated zero-shot on WildGuardMix.

SP-2 reached 79% accuracy, while both MultiMax and the last-token MLP reached 69%.

This matters because MultiMax had been stronger than SP-2 on the corresponding in-domain Nemotron test: 80.5% versus 78.2%. The ranking reversed under the dataset shift.

The transfer result is better treated as a robustness signal than as a second generalization thesis. It shows that the best in-domain monitor need not be the best transferred monitor. With only one explicit cross-corpus experiment, it does not establish that Speculative Probing will systematically generalize better under unfamiliar shifts.

More recursive probing is not automatically better

The prompt-length ablation also argues against buying additional computation by default.

SP-1 is the best Speculative Probing variant in 7 of 16 model-task cells and remains within 2.5 percentage points of the best prompt length in 14 of 16. Moving from one recursive query to two or five therefore does not produce monotonic improvement.

A second ablation compares the trainable linear classifier with a readout using the frozen language-model head. Differences stay within roughly ±2.5 percentage points and change direction across tasks. Neither readout consistently dominates.

For deployment, these ablations make the smallest configuration a reasonable starting point to validate—not because the paper proves SP-1 universally sufficient, but because additional recursive steps need to earn their extra computation on the actual monitoring task.

Existing speculative infrastructure changes the monitoring economics

For a team already running speculative decoding, the paper changes one architecture decision.

A safety or quality monitor does not necessarily need to be modeled as a separate inference service with its own large classifier and duplicated sequence processing. Several task-specific monitors could instead query representations already produced by the speculative head and share the cached prefix.

That could be relevant for prompt safety, contradiction detection, reasoning-pattern analytics, repetitive-loop detection, and failure-case collection. The affected decision is the incremental architecture and compute budget for monitoring, not whether monitoring itself is needed.

There is also a platform implication. If speculative heads can support monitoring as well as decoding acceleration, their representation quality may matter beyond tokens-per-second. The paper’s Qwen MTP probes generally perform better than the MiniCPM Eagle3 probes, but this comparison cannot identify why: head family, base model, and training history all change together. It would be premature to treat co-pretraining itself as the proven cause.

The low-cost claim has three production boundaries

First, the infrastructure must already exist. A model without a well-trained speculative-decoding head cannot obtain the same marginal-cost story without first adding or distilling one.

Second, the monitor still needs labels. The main tasks use training sets measured in thousands of examples. A new policy category, emerging abuse pattern, or unusual enterprise constraint therefore creates a data-operations problem even if inference becomes cheaper.

Third, deployment efficiency still needs measurement. The paper provides a clear computational argument and asymptotic comparisons, but not a broad wall-clock benchmark across production hardware, batching regimes, sequence lengths, and concurrent monitors.

There is also a governance consequence. Lowering the cost of behavioral classification can make pervasive monitoring easier to deploy. Systems using the technique still need purpose limits, privacy controls, and explicit handling of false positives; cheaper classification does not reduce the consequences of acting on a mistaken one.

Monitoring can become a reuse problem

The paper’s strongest idea is architectural rather than simply classificatory.

If a production system already spends computation to build a sequence-aware speculative representation, a monitoring pipeline may not need to reconstruct comparable information somewhere else. A small learned query can potentially reuse the state that is already there.

The benchmark results show that this reuse can preserve much of the accuracy of a stronger position-aware probe while avoiding a separate full-model classifier path. Whether that translates into a meaningful production advantage depends on three things the benchmark cannot decide for an operator: the quality of the deployed speculative head, the availability of representative labels, and the measured latency and robustness of the monitor under the system’s actual traffic.

Cognaptus: Automate the Present, Incubate the Future.