Roots of Understanding: When Transformers Try to Learn the Language of Numbers

Opening — Why this matters now

Modern AI models excel at human language, protein folding, and occasionally pretending to do mathematics. But ask them to infer the prime factorization of a number from a symbolic sequence, and they often blink politely. The paper Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees fileciteturn0file0 asks a sharper question: Can a transformer learn the grammar embedded in the integers themselves?

This is not about predicting markets or summarizing documents. It is about whether sequence models can internalize a deterministic, non‑empirical structure—one whose generative law we know down to the last symbol. In other words: if arithmetic is a language, can a transformer become literate?

Background — Context and prior art

Prime factorization is uniquely determined but computationally expensive. Prior machine‑learning attempts to model primes, gaps, or factorization patterns typically collapse under scale or rely heavily on handcrafted features. As the authors review, models fed raw integers behave as though they are eating static.

The novelty here is the encoding. Each integer is mapped to a rooted planar tree representing its full multiplicative structure, which is then translated into a Dyck word—a balanced binary string obeying strict combinatorial constraints. The resulting infinite sequence of such strings, denoted NT, is a symbolic text with:

• long‑range correlations,
• sublinear vocabulary growth,
• hierarchical motifs,
• directionality (some substrings appear, their reversals never do), and
• arithmetic meaning (e.g., primes always map to 10).

This converts the natural numbers into an exotic form of linguistics: deterministic, infinite, grammatically rich, and entirely non‑stochastic.

Why this representation matters for AI

Dyck words and trees carry structure rather than magnitude. Models cannot rely on numeric shortcuts or interpolation. Any success must come from learning the underlying grammar—for example, that square‑free numbers produce concatenations of prime‑shaped Dyck units.

For business readers: this is a stress test of structural generalization. If a transformer succeeds here, its ability to infer hidden rules in deterministic data may extend to compliance codices, financial audit trails, or algorithmic processes.

Analysis — What the paper does

The authors generate the first 10¹¹ integers’ tree encodings and tokenize them using a Byte‑Pair Encoding (BPE) vocabulary of size 64–1024. They train a GPT‑2–style decoder (12 layers, 12 heads, ~87M parameters) from scratch using:

1. Next‑Word Prediction (NWP): predict the next token given 1024 tokens of context.
2. Masked Language Modeling (MLM): fill in masked tokens representing Dyck‑word fragments.

A simple Markov chain serves as a deliberately weak baseline—if the transformer can’t beat it, the arithmetic language is effectively opaque to sequence models.

Key methodological choices

• Tokenization compresses recurring Dyck subpatterns, letting the transformer focus on grammatical structure.
• Training/validation/test splits preserve ordering: later segments of NT may contain patterns not seen earlier.
• Temperature (T) controls generation stochasticity; low T encourages deterministic prediction.

Findings — What the model actually learns

The results are a mix of intrigue and humility.

1. Transformers do learn nontrivial arithmetic structure

Word‑level accuracy (matching the true Dyck word at each position) outperforms the Markov chain across all temperatures. Performance peaks at T ≈ 0.1–0.3.

Precision/Recall for specific Dyck words

At the word level, the model reliably distinguishes recurring structural motifs. For example:

Interpretation: primes are harder because the local sequence around them is less predictable—a prime’s factorization carries no recursive substructure.

2. Distributional performance

The Kullback–Leibler divergence between real and generated word frequencies decreases significantly compared to the Markov baseline, indicating the model approximates the global grammar of NT.

3. Characteristic model errors

The paper’s confusion matrices reveal systematic misclassifications:

• The model frequently confuses primes with square‑free numbers—those whose Dyck words begin similarly.
• Very rare structural motifs are often under‑predicted.

In business terms: the model is good at learning repetitive hierarchical patterns but struggles with singular events—much like human auditors detecting routine patterns reliably yet hesitating on rare anomalies.

4. MLM results

MLM accuracy declines sharply as both masking rate and temperature increase. Still, at low temperature and moderate masking, token‑level accuracy exceeds 0.40—strong evidence the model reconstructs missing structure from context.

Implications — What this means beyond number theory

1. Transformers generalize structure, not just statistics

The model’s successes occur where the grammar of factorization exhibits local dependencies. Failures arise precisely at “prime boundaries,” where the next structure is uncorrelated with prior ones.

This distinction mirrors enterprise realities:

• Predictable compliance sequences (recurring workflows, audit trails) behave like composite numbers.
• Unpredictable events (fraud, edge‑case rule violations) behave like primes—requiring nonlocal reasoning.

2. Deterministic languages create a new testbed for AI assurance

Large models often succeed empirically without anyone understanding why. Arithmetic sequences offer the opposite: perfect interpretability of the data‑generating process. Evaluating models here could become a future gold standard for mechanistic interpretability, safety benchmarking, and AI compliance testing.

3. Structural reasoning remains a frontier

GPT‑2 learns the shallow grammar of factorization but not its global logic. Larger context windows and more expressive architectures may eventually reconstruct higher‑order arithmetic rules—raising intriguing questions about whether a sufficiently capable model could internalize number‑theoretic distributions.

Conclusion — The boundary of prediction is the boundary of understanding

Transformers can partially learn the “language of integers” when expressed as rooted‑tree Dyck words. They capture recursive motifs, frequency hierarchies, and local multiplicative structures—but stall at unpredictability intrinsic to arithmetic, especially primes.

For AI practitioners, the message is elegantly simple: sequence models learn structure where structure exists. They do not conjure it where the universe withholds it.

Cognaptus: Automate the Present, Incubate the Future.