Eight Attempts, Four Routes, One Policy

TL;DR for operators

A fixed budget of eight generated answers does not guarantee eight equally informative training experiences. Activation-Space Exploration (ASE), introduced by Zeng and colleagues,¹ changes how those answers are generated by temporarily steering a model’s internal representations along directions learned from verified successes and failures.

Against standard GRPO, ASE improves mathematical Mean@8 accuracy by 4.87 percentage points on Qwen3-1.7B and 1.87 points on Qwen3-8B, without increasing the eight-rollout training budget. Matched controls show that verifier-guided directions contribute beyond arbitrary steering. The intervention is used during training; the resulting model operates without it.

For training-system operators, the trade-off is measurable: approximately 6% lower training throughput, additional implementation complexity, and highly uneven gains outside mathematics. ASE merits evaluation where outcomes can be reliably verified, but its results do not establish broader agent reliability or universal improvements from activation steering.

Eight answers can still be a narrow search

Suppose a training team can afford eight attempts at each mathematical problem. The model generates eight solutions, an automated checker identifies successes, and those outcomes guide the next policy update.

If the attempts concentrate around similar high-probability behaviors, generating eight responses may provide less exploration than the budget suggests. Increasing the sample count costs additional computation. Increasing decoding randomness may produce more variation, but that variation need not expose more successful approaches.

The underlying training process is reinforcement learning with verifiable rewards (RLVR): generated solutions receive automatically checked outcomes, which are used to improve the model.

ASE changes a different part of this process. Instead of spending more samples, it generates the same number of responses under several temporarily modified internal conditions.

That gives model developers another design variable: the distribution of training experiences, not only their quantity.

The reported improvement is substantial enough to investigate. With eight rollouts per problem and four training epochs, mathematical Mean@8 on Qwen3-1.7B rises from 25.47% under GRPO to 30.34% under ASE-GRPO.

The harder question is what makes these interventions productive, since arbitrary steering also improves some results.

How verified outcomes become exploration routes

ASE begins with the model’s own training responses. For a given mathematical problem, it identifies correct and incorrect trajectories using binary verifier rewards. Groups without both outcomes cannot contribute to the direction-estimation process.

The method then extracts internal activations from those responses using the unsteered model. Comparing correct and incorrect responses to the same problem helps reduce variation caused by different prompts.

For each pair, ASE calculates a positive-minus-negative activation difference. It projects activations into a 64-dimensional principal-component space and uses K-means clustering to organize the resulting differences. The main configuration maintains three clusters.

Each cluster produces a steering direction by averaging its constituent activation differences in the original hidden-state space.

Because clusters can change across training iterations, ASE matches newly estimated directions to historical ones using cosine similarity and the Hungarian algorithm. Exponential moving-average smoothing reduces instability.

The resulting vector bank generates three temporary steering routes, accompanied by one unmodified route.

During rollout generation, a selected direction is added to the output of a Transformer block:

Here, $\mathbf{v}_{s_i}^{(t)}$ is the normalized direction assigned to the response and $\alpha$ controls its strength. The reported configuration uses $\alpha=0.5$ and intervenes at Transformer block 0.

Eight responses are allocated across the four routes.

Steered responses arise from modified token distributions. ASE therefore uses bounded token-level importance-sampling weights to account for differences between those proposal distributions and the unsteered policy. Verifier-derived advantages then enter a clipped GRPO objective.

There is still only one trainable actor policy. The steering vectors influence which trajectories are encountered, while policy optimization incorporates the verified outcomes into the unsteered model’s parameters.

After training, the vector bank can be discarded.

The accuracy gains are real within the tested comparisons, but uneven

The main experiments train Qwen3-1.7B-Base and Qwen3-8B-Base on 17,390 filtered DAPO-Math-17k problems. They compare methods under four training epochs and eight rollouts per problem.

GRPO → ASE-GRPO. Section 4.2, Tables 1–2. Values average three runs. Mathematical aggregates cover six benchmarks; out-of-distribution (OOD) aggregates cover three.

Mean@8 measures average accuracy across eight generated responses. Pass@8 measures whether at least one of those responses solves a problem.

On the smaller model, ASE raises both the average quality of sampled answers and the probability of finding at least one correct answer. Its mathematical Pass@8 gain is 6.35 percentage points.

On the larger model, mathematical Mean@8 improves by 1.87 points, while Pass@8 rises only 0.53 points. Out-of-distribution Mean@8 improves by just 0.03 points.

ASE also exceeds the reported DAPO, Dr.GRPO, GSPO, and RePO baselines on mathematical Mean@8 for both model sizes. But it does not dominate every individual benchmark or every Pass@8 comparison.

These are comparative benchmark improvements, not evidence of uniform reasoning gains.

Verifier guidance matters more than perturbation alone

The most informative ablation holds the clustered vector bank, routing procedure, and temporal smoothing fixed while replacing the learned steering directions.

On Qwen3-1.7B, mathematical Mean@8 reaches 27.76% with random directions, 27.56% with shuffled verifier labels, and 27.87% with reversed directions. Verifier-guided ASE reaches 30.34%.

All three alternatives improve upon the 25.47% GRPO reference, so some benefit may arise from the intervention machinery itself. Yet the matched comparison favors directions estimated from correct-versus-incorrect outcomes.

An additional appendix control uses unstructured random steering without clustering or smoothing. It gains only 0.65 Mean@8 points and loses 2.32 Pass@8 points against its own GRPO reference. Those first-evaluation results are separate from the main three-run aggregates.

Training diagnostics also show greater answer-level disagreement under ASE and persistent geometric separation among steering directions.

These observations are consistent with broader exploration, but different answers do not prove different reasoning strategies. The paper does not establish stable, human-interpretable meanings for the activation clusters.

The steering improves training without becoming a deployment dependency

A separate comparison asks whether applying steering directly to an already-trained model can produce similar gains.

For Qwen3-1.7B, inference-time steering increases standard GRPO’s mathematical Mean@8 from 25.47% to 25.81%. ASE-GRPO, evaluated without steering, reaches 30.34%.

The inference-time settings were selected from 84 tested configurations per checkpoint, making them exploratory best-case comparisons rather than evidence for a universally effective inference controller.

The larger improvement comes from using altered trajectories to update the policy during training.

ASE is also tested with Dr.GRPO and GSPO, producing mathematical Mean@8 improvements of 1.28 and 1.49 points, respectively, on the smaller model. These compatibility tests suggest the exploration mechanism is not exclusive to standard GRPO, although they do not establish equivalent benefits across optimizers or domains.

For deployment architecture, the practical advantage is clear: the tested policy does not require an additional steering system at inference.

The training bill and the generalization boundary

The intervention-free deployment design does not make exploration free.

ASE requires activation replay, vector-bank estimation, routed decoding, and importance correction. Against matched GRPO runs, training throughput declines by 6.29% on Qwen3-1.7B and 6.44% on Qwen3-8B. Mean step time rises by 0.91% and 7.19%, respectively.

Fixed rollout counts therefore do not imply identical training expenditure.

Configuration also matters. Appendix sensitivity tests show that increasing steering strength or the number of clusters does not monotonically improve performance. Moderate temporal smoothing often helps, but the sweeps vary parameters under different fixed settings and do not identify a joint optimum.

Several limitations directly affect adoption:

• Both evaluated models belong to the Qwen3 family, and training uses one mathematical dataset.
• Transformer block 0 is chosen heuristically, without a layer-selection sweep.
• MATH500 is repeatedly used for checkpoint selection, reducing its independence as an evaluation benchmark.
• Main benchmark results average three runs but lack comparison-specific confidence intervals or significance tests.
• Robustness to verifier errors, production task reliability, and financial return are not established.

Cognaptus inference: For teams training specialized reasoning models with reliable automated outcome checks, ASE is a candidate for a controlled training experiment. The adoption decision should compare independently evaluated task accuracy against actual GPU expenditure, engineering complexity, and stability across configurations.

A successful mathematical benchmark result should not substitute for verification on the organization’s own target tasks.

Better exploration must earn its additional computation

ASE demonstrates that the same number of training responses can produce different learning outcomes when their generation conditions are structured by verifier feedback.

Its strongest evidence combines matched rollout-budget improvements with direction-construction controls showing that outcome-guided steering performs better than arbitrary alternatives. The trained model retains the improvement without deployment-time activation intervention.

For operators, the next decision is empirical: determine whether that additional learning translates into enough independently verified task performance to justify the training overhead.

The method expands the design choices available for RLVR exploration. Its business value remains conditional on the tasks, verifiers, and compute budgets of the training system adopting it.

Cognaptus: Automate the Present, Incubate the Future.