Opening — Why this matters now
Human behavior is the final frontier of prediction. Chess—arguably the world’s most intensely instrumented strategy game—used to be about best moves. Today, it’s increasingly about human moves. As analytical tools migrate into coaching apps, anti-cheating systems, and personalized training platforms, understanding how different players actually behave (not how they ideally should) becomes commercially relevant.
The paper behind this article proposes a refreshingly low-tech yet surprisingly insightful approach: treat chess moves like language and train skill‑group‑specific n‑gram models to predict what humans will actually play. Rather than invoking trillion‑parameter transformers, it leans on the humble, fast, embarrassingly‑efficient n‑gram—reviving a classic NLP technique for a behavioral use case. fileciteturn0file0
Background — Context and prior art
Traditional chess engines optimize for correctness. They calculate the best move, evaluate deep trees, and operate under the sacred assumption that optimality is the only metric that matters.
The problem: humans don’t play optimal chess. They play their chess—shaped by habit, comfort, panic, fatigue, and that dodgy coffee they had before round three.
Prior research has explored chess as a language—e.g., fine‑tuning GPT‑2 on PGN data—but still through the lens of strategic correctness. What’s missing is a framework that accepts human inconsistency as a feature, not a bug.
Enter this paper’s proposition: split players by rating, build distinct language models for each skill group, and let the data speak about what humans at each level are statistically likely to do.
Analysis — What the paper actually does
The authors divide millions of Lichess games into seven rating bins (<=1000 up to 2250+) and pre‑process them into clean, tokenized move sequences. All annotations, engine evals, side variations, and move numbers are stripped out (page 3). The result: pure algebraic-notation streams—perfect for token-level modeling.
On these corpora, they train seven separate 5‑gram KenLM models, each representing the behavioral signature of a specific skill level.
Then comes the clever part:
- Model Selector: Given the first k moves of a game, identify which model (L1–L7) assigns the lowest cumulative surprisal → predicted skill level.
- Move Predictor: Use only that model to generate next‑move probabilities rather than taking a global max across all models.
This avoids the classic issue where a global probability ranking might accidentally select “L4‑like” moves for an L7 game—an incoherent mix that misrepresents real player tendencies.
Findings — Results in plain business terms
The results reveal a few striking behavioral patterns.
1. Early‑game behavior is the easiest to classify
Model selector accuracy:
| Level | Using 16 half‑moves | Using 100 half‑moves |
|---|---|---|
| L1 | 37.2% | 22.3% |
| L7 | 43.3% | 15.3% |
Both novice and expert players become less classifiable as the game progresses. Novices become random; experts become idiosyncratic.
In contrast, mid‑tier players (L2–L5) become more predictable when more information is available.
2. Top‑1 prediction: modest gains but consistent
Selector‑based predictions outperform the benchmark by up to 6.6%. Accuracy dips near the 50‑move mark—precisely where humans face the combinatorial explosion of middle‑game choices.
3. Top‑3 prediction: where the method shines
Top‑3 prediction sees accuracy improvements of up to 39.1% (page 8). This doesn’t merely show better modeling; it validates an intuitive truth:
Humans often consider several plausible moves. Narrowing to only one overstates the determinism of human decision-making.
Visualization — Conceptual Framework
| Component | Purpose | Business Analogy |
|---|---|---|
| Skill-specific n‑gram LM | Captures human patterns at each skill level | Customer segmentation |
| Model Selector | Identifies which segment the user belongs to | Lead scoring engine |
| Move Predictor | Generates next move probabilities | Personalized recommendation system |
This is essentially behavioral segmentation + sequence prediction, repurposed for chess.
Implications — Why this matters beyond chess
Though framed around chess, the implications are broader:
- Human-centric AI: Systems that differentiate user competence are more adaptive, more personalized, and less brittle.
- Anti-cheating systems: Skill‑aware surprisal scoring can detect moves inconsistent with a player’s typical profile.
- Training platforms: Coaches could benchmark a student’s “behavioral drift” over time.
- Real-time analysis tools: The use of KenLM and simple n‑gram LMs shows that ultra‑low‑latency inference remains viable when accuracy is framed around human realism rather than engine optimality.
In short: predicting humans is a different optimization problem than predicting truth.
Conclusion
This paper delivers a simple but sharp message: When analyzing humans, don’t force an optimality framework onto a variability problem. By modeling skill‑specific behavioral patterns using n‑gram language models, the authors surface something engines routinely overlook—how humans actually play.
It’s fast, it’s interpretable, and it reminds us that older NLP tools still have bite when the goal is human realism.
Cognaptus: Automate the Present, Incubate the Future.