Adversaries, Slices, and the Art of Teaching LLMs to Think

A math tutor does not wait until the end of a two-page solution, circle the final answer, and say “wrong.”

At least, not a good one.

The useful tutor interrupts earlier. This line follows. That parity condition does not. This factorization is legal, but the conclusion you drew from it is not. The feedback is local, not theatrical. It tells the student where the reasoning began to rot, before the final answer becomes merely the visible corpse.

Most reinforcement learning for reasoning models still behaves more like the lazy tutor. It rewards the final answer and hopes the thinking process improves as a side effect. Sometimes it does. Often it produces models that are better at arriving somewhere, without being much better at knowing which steps deserved trust along the way.

The paper behind this article, Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning, proposes a different training loop: pair a reasoner with a discriminator, cut reasoning chains into manageable slices, and let the critic learn alongside the model it judges.¹ The headline is improved benchmark accuracy. The more interesting story is how the reward signal is built.

This is not just “use another model as a judge.” We already have plenty of those, some useful, some very confident about nonsense. Nor is it mainly an inference-time debate trick where models argue until a winner emerges. GAR is a training-time co-evolution system. The reasoner learns to produce better reasoning; the discriminator learns to inspect the reasoner’s current behavior; and the reasoner uses those slice-level judgments as dense process feedback. At inference time, the discriminator disappears. The student takes the exam alone.

That distinction matters. It is the difference between hiring a reviewer to comment on every report forever and training a writer whose internal habits improve because the reviewer was present during apprenticeship. One is operating expense. The other is capability formation. In AI, as usual, the expensive part is pretending those are the same thing.

The real problem is sparse praise, not lack of chain-of-thought theater

The modern reasoning model already knows how to produce long explanations. That is not the bottleneck. The bottleneck is knowing which parts of the explanation deserve reinforcement.

Outcome-based reinforcement learning gives a model a simple signal: final answer correct or final answer wrong. This works well when the target is easy to verify and the reasoning path is short enough that credit assignment is not too mysterious. But mathematical reasoning, code generation, proof search, financial analysis, and policy interpretation rarely fail in a single clean location. They fail in the middle: a hidden assumption, a skipped case, a misleading analogy, a calculation that remains syntactically beautiful while being numerically dead.

Process Reward Models try to solve this by judging intermediate steps. Their appeal is obvious: if each step can be scored, the model receives richer feedback than a binary final grade. Their weakness is equally obvious: high-quality step labels are expensive, subjective, and hard to scale. Automated judges reduce cost, but fixed critics can become stale. Once the reasoner changes, the old critic may mis-score new reasoning patterns, rewarding polished wrongness or punishing unfamiliar but valid solution paths.

GAR’s answer is to avoid treating the critic as a frozen oracle. The discriminator is itself trained. It is not merely a passive evaluator attached to an RL loop; it is a moving adversary whose job is to detect flawed slices and distinguish generated reasoning from reference reasoning. The reasoner, in turn, learns from the discriminator’s slice-level rewards plus the usual exact-match reward on final answers.

So the central move is not “more reasoning.” It is better supervision over reasoning.

That is why the paper is best read mechanism-first. The results matter, but the design explains why those results are plausible.

GAR cuts reasoning into slices because whole-chain judgment is too blunt

A full chain of thought can span thousands of tokens. Asking a model to judge the entire thing is like asking an auditor to review a multinational tax filing in one pass and return a binary verdict. Technically possible. Operationally silly.

GAR instead partitions a reasoning trajectory into shorter, logically coherent slices. The paper describes a rule-based segmentation strategy using delimiters and discourse markers, then merging adjacent segments until they form semantically meaningful chunks within a target token range. The discriminator evaluates each slice for logical soundness and returns a binary judgment with a concise rationale.

The slice-level reward is then aggregated, in the main design, as the mean of the slice-level scores. The reasoner’s reward combines this process signal with exact-match grading of the final answer.

In simplified terms:

where $R_{\text{exact}}$ reflects whether the final answer matches the ground truth, and $R_{\text{slice}}$ reflects the discriminator’s average judgment over reasoning slices.

This matters because wrong answers are not equally wrong. A model that follows a valid setup but makes a late arithmetic slip should not receive the same training signal as a model that invents a false theorem in line two and then confidently sprints into the swamp. Exact-match grading treats both as failures. Slice-level scoring can distinguish them.

The paper’s examples show the discriminator identifying exactly this kind of local failure. In one case, a reasoning slice correctly handles much of a number-theory argument but then overclaims that a factor pair exists under conditions that do not guarantee it. The discriminator marks the slice “NO” and explains the missing requirement. In another case, a model enumerates permutations and prefix sums correctly; the discriminator marks the slice “YES” while noting that the reasoning has not yet proven the broader minimum claim.

That second detail is important. A good critic does not need to pretend every incomplete argument is wrong. It needs to know whether the slice under inspection is locally sound. This is closer to real review practice than the usual “correct answer equals good reasoning” superstition.

The adversarial part is not drama; it is reward maintenance

The word “adversarial” tends to attract two bad interpretations. One is cinematic: two AI agents fighting in a digital courtroom. The other is vague: adversarial equals robust, therefore good, please clap.

GAR is more specific. The discriminator receives two complementary rewards.

First, it receives an alignment reward, encouraging its slice-level judgments to agree with answer-level correctness. If reasoning leads to a correct answer, its slices are more likely to be sound; if it leads to a wrong answer, something probably failed. This assumption is useful but imperfect, because models can sometimes reach correct answers through shaky reasoning or wrong answers through mostly correct work with a late slip.

Second, the discriminator receives a discriminative reward inspired by GAN-style objectives: distinguish generated reasoning slices from reference reasoning slices. This encourages the discriminator to remain sensitive to differences between model-produced reasoning and stronger reference traces.

The reasoner is then trained with Group Relative Policy Optimization, using the blended final-answer and discriminator-based process reward. The two models update together. That on-policy co-training is the conceptual hinge.

A fixed critic is a photograph. GAR wants a critic that remains alive while the reasoner changes.

The result is a training loop where the model is not only rewarded for being right. It is rewarded for producing intermediate reasoning that a co-evolving critic judges as sound.

That is a subtle but useful shift. It turns reasoning improvement into a feedback architecture problem rather than a motivational slogan.

The main evidence: gains are broad, but largest where reasoning is hard

The main benchmark evidence comes from seven mathematical reasoning benchmarks, evaluated with Pass@1 accuracy averaged over 30 runs per benchmark. The authors test GAR on two DeepSeek-R1-Distill backbones: Qwen-7B and Llama-8B.

The headline results are strongest on difficult benchmarks such as AIME and LiveMathBench-Hard.

The pattern is more informative than any single number.

On Qwen-7B, AIME24 improves from 54.0 to 61.3, a gain of 7.3 points. AIME25 improves by 6.3 points. LiveMathBench-Hard rises by 6.5 points. On Llama-8B, AIME24 improves by 10.0 points, AIME25 by 5.9, and AMC23 by 5.5.

The easier or more saturated benchmarks show smaller improvements. MATH500 for Qwen moves only from 94.3 to 94.8. That is not a failure; it is what one should expect when the baseline is already near the ceiling. The gains concentrate where sparse final-answer rewards are least informative and where intermediate reasoning quality has more room to matter.

This makes the paper more credible, not less. If the method claimed giant gains everywhere, including on already saturated tasks, one would have to start checking for evaluation magic. Here the improvement profile fits the mechanism: dense process supervision helps most when reasoning depth and error localization matter.

The ablations show GAR is not just “standard RL plus vibes”

The ablation table is the part business readers should not skip. It separates the mechanism into components and asks which pieces actually contribute.

Starting from DeepSeek-R1-Distill-Qwen-7B, standard RL with exact-match grading improves AIME24 from 54.0 to 56.3 and AIME25 from 38.0 to 40.7. Adding a fixed standard critic barely changes the picture: 56.7 on AIME24 and 40.4 on AIME25. That is a useful warning. Attaching a critic does not automatically make training smarter. It may simply add noise with better branding.

The slice-specific GAR discriminator changes the story. A fixed GAR-style discriminator reaches 58.6 on AIME24 and 42.0 on AIME25. A trainable discriminator with only one of the discriminator-side reward terms improves further. Combining exact match, judger score, alignment reward, and discriminator reward reaches the final 61.3 and 44.3.

The ablations support a precise claim: GAR’s gains come from the combination of slice-level evaluation, discriminator training, and joint on-policy updates. Remove those details, and the method becomes much less interesting.

This is also where the “not just another verifier” misconception should die quietly. A verifier checks. GAR trains through checking. Those are adjacent ideas, not identical ones.

The compute story is less magical than the accuracy story, which is good

Dense process feedback sounds expensive because it is. Evaluating every reasoning slice with another language model can become computationally absurd very quickly. The authors address this with a truncated discriminator response format: brief analysis, binary judgment, concise rationale, capped at 128 tokens.

The rollout speed comparison is revealing.

This is not “free accuracy.” GAR with truncation still takes longer than standard RL. But the additional cost is moderate in the reported setting, while the untruncated version more than doubles training time without improving accuracy. That makes the truncation result a practical implementation detail, not a decorative appendix note.

For enterprises, this is the difference between a research idea and an engineering candidate. A method that improves reasoning but doubles training cost for no additional benefit belongs in a lab notebook. A method that uses concise critic outputs and preserves most of the efficiency story has a more plausible path into fine-tuning pipelines.

The quiet lesson: process supervision needs a budget discipline. Long rationales from critics may feel more transparent, but during training they can become expensive theater. The verdict must be good enough, localized enough, and cheap enough. Nobody gets a medal for burning H100 hours on beautiful explanations the optimizer barely needs.

The entropy analysis suggests better calibration, not just better scoring

The paper also analyzes entropy behavior. This is not the main evidence in the same way as the benchmark table, but it serves an important diagnostic purpose.

A common concern in RL-for-reasoning is entropy collapse: the model becomes more accurate by becoming less exploratory, narrower, and potentially worse calibrated. GAR’s authors report that Qwen-7B with GAR improves AIME24 accuracy while maintaining a comparable overall mean-entropy distribution to the baseline: 5.20% versus 5.27%. They also report fewer extreme failures among wrong answers and a more nuanced pattern after removing zero-entropy tokens.

The interpretation is that GAR encourages selective entropy. The model becomes more decisive on deterministic spans while retaining exploration on decision-critical tokens. In plain English: less wandering where the logic is settled, more flexibility where genuine reasoning choices remain.

For business use, this matters because reliability is not merely accuracy. A model that is confidently wrong in brittle ways is harder to govern than a model that exposes uncertainty in the right places. If token- or slice-level uncertainty can help trigger self-checks, adaptive sampling, or escalation to human review, then entropy diagnostics become operational signals.

But this should be read carefully. The entropy analysis supports a calibration story for the tested math setting. It does not prove that GAR-trained models will be well calibrated in legal review, financial forecasting, medical triage, or enterprise workflow automation. Different domains produce different failure modes. A math model’s entropy behavior is a clue, not a compliance certificate.

Partial-trace training is the paper’s underrated business idea

One of the more interesting experiments is not the largest benchmark table. It is the partial-trace setting.

Standard RL post-training often depends on full reasoning chains and verifiable final answers. That is manageable for math problems with known answers or coding problems with test cases. It becomes awkward when outputs are open-ended, when verification is delayed, or when the final answer is not easily reducible to pass/fail.

GAR tests a partial-trace variant: stop after three reasoning slices, evaluate those slices with the discriminator, and train without requiring a full chain of thought or final-answer reward. The reported result: 57.7 on AIME24 in 6 hours, compared with standard RL at 56.3 in 16 hours.

This is labeled by the authors as a future direction, and that is the right level of confidence. Still, it is the part that should make enterprise AI teams pay attention.

Many business tasks do not have clean final-answer labels. A compliance memo, a due diligence summary, a financial risk explanation, or an internal audit finding may be judged by experts, but not by a simple exact-match script. If training can use partial reasoning traces and critic feedback before complete task outputs exist, then process supervision becomes more adaptable.

Cognaptus inference: the practical value may be less “train models to ace math contests” and more “build internal evaluators that reward good reasoning behavior before downstream outcomes are fully measurable.”

That is an inference, not a result directly proven by the paper. The paper demonstrates partial-trace training in a math context. It does not demonstrate enterprise document review, strategic planning, or regulated decision support. But the mechanism points in that direction.

The coding results are promising, but they are secondary evidence

The appendix evaluates GAR on code generation. The setup uses CodeForces-CoT data, approximate ground-truth reasoning traces produced by DeepSeek-R1, and a reward function combining public test-case performance, code-format reward, and critic reward.

The results for DS-R1-Distill-Qwen-7B are:

The LiveCodeBench gain is meaningful: +6.2 points. HumanEval and HumanEval+ improve more modestly.

This appendix result is best treated as comparison with another reasoning-heavy domain, not as the paper’s central proof. Coding tasks involve executability, format constraints, hidden tests, and problem-solving traces that differ from mathematical reasoning. The fact that GAR helps here suggests the framework is not purely math-bound. But the evidence is still narrower than a broad “GAR improves software engineering” claim.

For AI teams building coding assistants, the useful takeaway is more specific: critic-shaped process rewards may help train models to produce better solution paths, especially when combined with executable feedback. It does not replace test suites. It may make the training signal around test suites less sparse.

That is less sexy than “AI learns to code like a human.” It is also less likely to embarrass us later.

Slice design is not cosmetic; it controls the critic’s field of vision

The appendix on segmentation design is especially useful because it explains why “just split the chain” is not a trivial instruction.

The authors compare their rule-based slice segmentation method with pure fixed-length token windows and LLM-based semantic segmentation. Fixed windows are cheaper but can cut through coherent reasoning. LLM-based segmentation is more semantically elegant but slower. The reported results:

This is a classic engineering trade-off. LLM-based segmentation slightly outperforms the rule-based method, but at much higher training cost. The authors choose the rule-based method because it captures most of the accuracy benefit with far better efficiency.

They also test slice length. Performance is highest and most stable around 320–560 tokens. Very short slices often lack enough reasoning content for meaningful feedback. Very long slices tend to contain at least one flaw, which makes too many slices negative and reduces label diversity. The reported AIME24 scores range from 57.4 at 160 tokens to 61.5 at 480 tokens, then decline to 56.8 at 1440 tokens.

That finding has a broader lesson. In process evaluation, granularity is a design variable. Too small, and the critic sees fragments without meaning. Too large, and the critic sees a mess. The business translation is straightforward: if you build internal AI evaluators for reasoning-heavy workflows, the unit of review matters. A paragraph, a calculation block, a legal issue, a code function, and a policy clause are not interchangeable slices.

The critic can only judge what the system chooses to show it.

Distillation shows the discriminator can shape reasoning style

The paper also includes a reasoning-style distillation experiment. The authors train a discriminator to distinguish Gemini-style and DeepSeek-style reasoning trajectories, then use the GAR framework to make the reasoner’s output harder to distinguish from Gemini reasoning. Human experts’ success rate in distinguishing generated reasoning from Gemini reasoning drops from 82.3% without GAR to 55.9% with GAR, close to a 50% random-guess baseline.

This is an exploratory extension, not the main result. It does not prove that the model becomes better at reasoning because it sounds more like Gemini. Style and substance are not the same thing, a fact worth tattooing on every evaluation dashboard.

But the experiment shows that the discriminator is a programmable lens. It can reward not only correctness but also patterns of reasoning. That opens a path toward preference alignment, house-style reasoning, domain-specific explanation norms, or expert-like procedural habits.

For businesses, this is both useful and dangerous. Useful, because organizations often need models to reason in ways that fit professional standards: cite assumptions, separate facts from inference, flag uncertainty, document alternatives. Dangerous, because style alignment can become plausibility laundering. A model that reasons in the tone of an expert is not necessarily thinking like one.

The right product lesson is not “make the model sound like your best analyst.” It is “define which reasoning behaviors matter, then train critics to reward those behaviors without mistaking surface imitation for competence.”

What GAR directly shows, what Cognaptus infers, and what remains uncertain

The paper directly shows that GAR improves DeepSeek-R1-Distill Qwen-7B and Llama-8B on a set of math benchmarks, with additional positive evidence on coding tasks. It shows through ablations that slice-level GAR discriminators, trainable critic rewards, and joint updates each contribute to the final result. It shows that truncated discriminator outputs can preserve accuracy while greatly reducing training time compared with untruncated critic reasoning. It shows promising partial-trace and reasoning-style distillation extensions.

Cognaptus infers that the broader business relevance lies in process-level quality control. Many enterprise AI failures are not final-answer failures at first glance. They are reasoning-process failures: a missed exception in a compliance workflow, an unsupported assumption in a market report, a faulty dependency in a code migration, or a document summary that preserves facts while breaking causal structure.

GAR suggests one possible training architecture for such problems: define meaningful slices, train a critic to judge them, keep the critic aligned with the model’s evolving behavior, and use dense process rewards rather than waiting for final outcomes.

What remains uncertain is the transfer path. The paper’s strongest evidence is in math. Coding support is encouraging but still limited. Enterprise reasoning has messier labels, noisier reference answers, and higher disagreement among experts. The discriminator may inherit domain bias. Slice labels may be expensive to validate. Reward hacking remains possible, especially if the model learns to produce critic-friendly reasoning that looks rigorous without improving underlying truthfulness.

So the responsible business reading is neither “deploy this tomorrow” nor “interesting but academic.” It is: if your AI workflow depends on reasoning quality, start designing the review units and critic signals now. The exact GAR implementation may not be your production method, but the architectural lesson is difficult to ignore.

The ROI is in cheaper diagnosis, not just higher benchmark scores

For firms fine-tuning LLMs, the obvious value is accuracy. Better AIME, better LiveCodeBench, better benchmark tables. Fine. Benchmarks are the industry’s favorite scoreboard, and sometimes scoreboards are useful.

But the more durable ROI may come from cheaper diagnosis.

Outcome-only training tells you whether the model failed. Slice-level process supervision can tell you where it failed. That difference compounds. It improves training efficiency, debugging, evaluation design, and human review. It also creates a more inspectable development loop: instead of asking whether a model is “good at reasoning,” teams can ask which classes of reasoning slices fail most often.

For business process automation, this maps naturally onto several workflows:

This is where GAR’s mechanism becomes more interesting than its exact benchmark numbers. Enterprise AI does not merely need models that answer. It needs systems that can be trained, audited, and improved around the places where reasoning breaks.

A discriminator is not a governance program. But a well-designed discriminator can become part of one.

The boundary: GAR improves the training loop, not the laws of truth

The authors are clear about remaining limitations. Balancing discriminator reasoning depth with compute efficiency remains difficult. The paper’s analysis–score–rationale format is a practical compromise, not a final solution. The current method aggregates slice rewards into a trajectory-level average, which can dilute local credit assignment and increase variance. Better use of slice-wise information may improve stability.

There are also boundaries beyond the paper’s own limitation section.

First, the discriminator’s judgment is only as good as its training signal and task framing. If the reference traces are biased, shallow, or stylistically narrow, the discriminator can reward the wrong habits.

Second, correctness in math is cleaner than correctness in business reasoning. A mathematical proof has stricter validity conditions than a market entry memo. In business contexts, expert disagreement is not noise to eliminate; sometimes it is the point.

Third, process feedback can be gamed. A model may learn to produce reasoning that satisfies the critic’s rubric without improving real-world reliability. GAR’s adversarial co-training is partly designed to reduce this, but no reward system deserves blind trust. Reward hacking is not a bug in one paper; it is the unpaid intern living inside every optimization objective.

Fourth, inference-time deployment uses only the reasoner. That is efficient, but it also means the discriminator’s direct oversight is absent when the model is used. If a business needs runtime assurance, it may still need separate evaluators, monitors, or human review for high-stakes outputs.

These limitations do not weaken the paper’s contribution. They define its operating envelope.

Teaching models to think means teaching critics to see

GAR’s core idea is simple enough to state without ceremony: do not train reasoning models only by applauding correct final answers. Train them with critics that can see the reasoning process in manageable pieces, and let those critics evolve as the model improves.

The paper’s strongest contribution is not that it adds another evaluator to the crowded shelf of AI judges. It shows a way to make the evaluator part of the learning dynamics: slice the reasoning, score the process, update the reasoner, update the discriminator, repeat.

The evidence supports meaningful gains on mathematical reasoning, smaller but positive gains on coding tasks, and promising extensions in partial-trace training and reasoning-style distillation. The business interpretation is narrower but more valuable: process-level supervision could become a practical foundation for training AI systems that do not merely produce answers, but learn where their own reasoning deserves correction.

That is the uncomfortable part for organizations rushing to automate knowledge work. If you cannot define what good reasoning looks like in slices, you probably cannot govern it at scale. You can still buy a larger model, of course. The industry will happily sell you one. It may even produce longer explanations.

But longer explanations are not the same as better thinking.

GAR is a reminder that teaching machines to reason may depend less on making them talk more, and more on building critics that know when to interrupt.

Cognaptus: Automate the Present, Incubate the Future.