The Vanishing Quotient: When More Training Erases Arithmetic Skills

TL;DR for operators

A compact AI model can learn a numerical procedure, perform it successfully in testing, and then lose that capability during further training. Sourabh Kasliwal’s study of tiny Transformers¹ illustrates this sharply: a model correctly solved 86.67% of 300 division problems with single-digit divisors, but its accuracy on that same subset fell to 0% after another 20,000 mathematical training steps.

The research identifies three distinct engineering constraints. First, arithmetic procedures must be decomposed into intermediate operations the model can manage. Second, training windows must contain enough supervised tokens to support learning; inefficient padding was associated with an accuracy collapse from 40% to 1%. Third, newly acquired skills require explicit retention testing.

The strongest model achieved only 35.85% overall accuracy on a 4,000-problem benchmark, despite much better performance on selected tasks. For organizations fine-tuning compact models, the lesson is to treat task acquisition, input-range generalization, and capability retention as separate release criteria. The paper demonstrates these issues in synthetic arithmetic, not production business calculations.

A model that improves at addition can forget division

Imagine a team preparing a compact language model to handle repetitive numerical operations. An intermediate checkpoint passes most of its division tests. The team continues training, expecting the model to become more reliable before deployment.

The next checkpoint improves at addition but can no longer perform the division task it previously handled.

This is what happened in Kasliwal’s experiments. A Modern Dense Transformer trained for 30,000 mathematical steps answered 260 of 300 held-out single-digit-divisor problems correctly. At 50,000 steps, it answered none correctly. Meanwhile, its accuracy on familiar addition problems increased from 93.39% to 99.22%.

The paper reports a highly significant difference in division outcomes using Fisher’s exact test (p < 10⁻¹⁵). The result establishes substantial performance deterioration between the evaluated checkpoints, not a universal tendency for further training to damage arithmetic.

The authors associate the decline with the training curriculum wrapping back to earlier addition and subtraction examples. Those examples displaced the division-focused training distribution, apparently weakening a recently acquired capability. The study does not experimentally isolate this explanation from every other possible source of training interference.

For a team selecting a deployment checkpoint, the consequence is immediate: improving the aggregate training objective cannot substitute for testing whether previously working operations still work.

Long division improves when intermediate calculations become manageable

The study investigates whether small autoregressive Transformers can execute deterministic arithmetic by producing intermediate calculations before their final answers.

Its common backbone has six layers, six attention heads, hidden dimension 384, and a 256-token context. The dense configurations contain approximately 49.3 million parameters including embeddings. The experiments cover addition, subtraction, multiplication, and division.

Rather than supervising only final answers, the researchers construct algorithmic scratchpads: explicit sequences representing the intermediate work needed to complete a calculation.

Addition is represented through place-value decomposition. Subtraction tracks borrowing. Multiplication generates partial products and aligns their decimal positions. Division constructs quotient digits sequentially, updating remainders and bringing down the next digit.

The most consequential change concerns long division.

Earlier configurations attempted partial-quotient estimation. The later approach expresses division as repeated, smaller digit-level decisions. It is paired with a four-stage curriculum that introduces additive foundations, multiplication facts, multi-digit multiplication, and finally long division, with some rehearsal of earlier tasks.

On the 100-problem development set, this combination produced division accuracy of 56% for Modern Dense and 60% for sparse mixture-of-experts (MoE). The larger benchmark provided stronger evidence for a narrower capability: 86.67% accuracy on single-digit-divisor problems for Modern Dense.

However, accuracy with multi-digit divisors was only 9.43%. Across all 1,000 division problems, the same checkpoint achieved 32.60%.

This difference matters more than the headline figure. The model acquired a relatively successful procedure under particular input conditions, but had not established broadly reliable long division.

Nor can the experiment identify the independent benefit of the new scratchpad. The deterministic algorithm and staged curriculum changed together. Their combined configuration performed well on the single-digit subset; their separate causal contributions remain unresolved.

Training-token density can matter before model capacity

A second finding concerns something less visible than model architecture: how arithmetic examples occupy the training context.

In one data-engineering comparison, continuously streamed arithmetic training reached 40% overall accuracy. Replacing it with individual problems padded to fixed-length windows reduced accuracy to 1%.

The padded configuration left approximately 82.8% of token positions without supervised loss. Across 5,000 training steps, active training tokens fell from 5.12 million to approximately 900,000.

The model was still performing optimizer steps, but each step delivered substantially less relevant learning signal.

The researchers addressed this inefficiency through sequence packing: placing multiple complete arithmetic examples into a fixed 256-token context. They also used prompt masking, assigning ignored labels to question tokens while keeping intermediate reasoning, final answers, and stopping tokens supervised.

This achieved approximately 97% active-token density.

The 40%-versus-1% result compares streaming with padded instruction training; it is not a direct measured accuracy improvement from packing itself. Packing addresses the identified supervision-density problem without establishing that every model or dataset will obtain the same accuracy gain.

Other comparisons reinforce the need to diagnose training design before simply increasing compute. After 5,000 arithmetic fine-tuning steps, Modern Dense achieved 33% accuracy with 20,000 linguistic pretraining steps but only 19% with 50,000 pretraining steps.

For a fine-tuning team, counting training steps alone is therefore an incomplete efficiency metric. The number and distribution of supervised output tokens also deserve inspection.

Multiplication exposes the difference between showing work and simplifying work

Explicit intermediate calculations are not automatically sufficient.

The paper’s multiplication procedure uses FOIL-style partial products. The model generates component products and accounts for decimal-place zeros, then must aggregate several multi-digit terms into a final answer.

An inspected failure trace shows correct individual partial products but an incorrect final sum.

The distinction is computational. Producing several simpler terms does not eliminate the difficulty of combining them. The final aggregation can remain a complex operation even when every preceding component is correct.

The extended-training experiments support this concern. After 100,000 mathematical training steps, both Modern Dense and sparse MoE reached perfect addition and subtraction scores on the small development evaluation, yet multiplication and division remained at 8% under the extended FOIL configurations.

The trace inspection helps explain a possible bottleneck; it does not prove that all multiplication failures share that mechanism.

The authors propose replacing simultaneous multi-term aggregation with sequential pairwise summation. This would break the remaining difficult step into smaller additions, but it was not implemented or evaluated.

For numerical-agent designers, the implication is to inspect whether an intermediate representation actually reduces every difficult computation or merely makes the unfinished complexity visible.

The larger benchmark reveals what small tests conceal

The 100-problem development evaluation was useful for comparing configurations and selecting checkpoints. The separate 4,000-problem benchmark examines four operations across different operand sizes and training checkpoints.

The results below show why checkpoint selection requires more than one headline score.

Source: Section 7.3, Table 5. Accuracy measures exact numerical answers generated using temperature-1.0 sampling.

At 30,000 steps, Modern Dense correctly answered 1,434 of 4,000 problems. The corresponding sparse MoE checkpoint achieved 33.92% overall, despite slightly outperforming Modern Dense on the full division subset.

The architecture comparisons also deserve careful interpretation. On the smaller 5,000-step development experiment, Modern Dense reached 33% overall versus 5% for baseline GPT-2, while sparse MoE reached 36%. But the modern components were bundled, and MoE had both more total parameters and higher active computational cost. The results do not isolate the effect of individual components or demonstrate a compute-matched MoE advantage.

The length-generalization failure is particularly striking. Every evaluated architecture-checkpoint combination answered zero of 486 unseen four-digit addition problems correctly. Four-digit subtraction performed only slightly better, ranging from 0.62% to 1.86%.

This does not conclusively establish an intrinsic computational-depth limitation. The GPT-2 byte-pair tokenizer represents shorter and longer numerical strings differently, potentially confounding the extrapolation test.

Nevertheless, the observed outcome is sufficient to challenge a deployment assumption: high accuracy on familiar operand lengths does not establish reliability when the numerical input range expands.

Four separate checks before releasing a numerical model

Cognaptus interprets the results as a case for four distinct release checks when organizations develop compact models for repetitive numerical workflows.

Training-data efficiency. Examine how many tokens actually contribute to the supervised objective. Padding rates, prompt masking, and packing efficiency should be measured before increasing training duration or model size.

Computational decomposition. Test the intermediate steps as well as the final answer. In particular, identify aggregation operations that remain difficult after the model generates apparently correct partial work.

Input-range generalization. Evaluate each operation across the operand lengths, divisor sizes, and numerical formats the deployed workflow may encounter. A strong subset score should not be treated as coverage of a broader task family.

Checkpoint retention. Maintain a fixed regression suite across previously learned operations. A newly trained checkpoint should not replace an existing one solely because its recent-task accuracy improved.

These are engineering recommendations inferred from the paper, not experimentally validated financial-workflow improvements. For applications requiring exact arithmetic, independently verified deterministic calculations remain an appropriate safeguard against unsupported model-generated numerical outputs.

What the experiments have not established

The evidence comes from synthetic arithmetic, with single-seed training and stochastic decoding. The division benchmark also uses problems constructed to have exact integer quotients, leaving non-exact division untested.

The paper does not separately identify curriculum effects from the deterministic division algorithm. Nor does it establish whether the same results persist across training seeds, alternative tokenizers, or independently tuned architectural configurations.

Finally, the proposed permanent rehearsal buffer is a plausible response to forgetting, but its effectiveness was not experimentally tested. The same applies to pairwise multiplication aggregation.

These limitations restrict causal attribution and production generalization without diminishing the observed checkpoint failures.

The next checkpoint must prove it has kept the last one’s skills

This research shows that a tiny Transformer can acquire a surprisingly strong arithmetic subskill when training and intermediate computation are organized appropriately. It also shows how narrow that success can be.

The division result matters because it demonstrates measurable learning under constrained model capacity. The subsequent collapse matters because it exposes the fragility of that learning under continued training.

For compact AI development, the standard should therefore extend beyond whether a model learns a procedure. It must establish where the procedure works, which computational steps remain unreliable, and whether the capability survives the next training update.

Cognaptus: Automate the Present, Incubate the Future.