LLM Training

TL;DR for operators A model generates an incorrect answer. An automated verifier can reject it but cannot explain the mistake. ReTeach1 shows how that failure can become training material without a worked solution or a larger external teacher. The model’s teacher revisits unsuccessful attempts, reflects on possible errors, and retries. But the successful retry is not simply copied into the student’s training targets. Instead, the teacher uses its additional reasoning context to produce better-informed token predictions along the student’s original response. The student learns from those predictions without needing that context itself. On SciKnowEval-Physics, explicit reflection achieves 64.24% accuracy, versus 62.91% with retries but no reflection (Section 4.3, Figure 4). ...

TL;DR for operators A pre-training contamination test can produce a clean-looking power law without producing a reusable contamination rule. In controlled OLMo-style runs, increasing the poison fraction approximately follows a power law in relative clean-validation perplexity degradation, but the fitted relationship changes over training and with model size. Halder, Dey, and Pehlevan’s A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents1 explains why such movement may be structural rather than measurement noise. ...

TL;DR for operators LLM reinforcement learning spends a large share of its wall-clock budget generating responses. Speculative decoding can reduce that cost, but it normally assumes that a smaller predictor is already capable of proposing useful tokens before the main model verifies them. GrowMTP1 tests whether that capability can instead be learned during the RL workload itself. On Qwen3-4B, a randomly initialized five-token draft head reached an acceptance length of 2.91 after 500 mathematical-reasoning RL steps. Rollout generation became 2.13x faster, and the complete RL step—including draft-head training—became 1.60x faster. ...

TL;DR for operators Synthetic data should have a defined job before anyone scales its production. For model training, the relevant test is whether generated examples add nonredundant learning signal and improve held-out performance without unacceptable regressions. For consumer research, the test changes: statistically diverse text is not enough if the business claim concerns what real customers believe. ...

TL;DR for operators A pretraining-data pipeline receives millions of uneven records. Some are unusable, some contain removable noise, some need structural repair or added explanation, and some are already valuable enough that further processing may damage them. The operational problem is therefore not how to apply more cleaning, but how to decide which intervention—if any—each example needs. ...

Synthetic and Sensibility: Why More Data Needs a Control Stack Synthetic data has become the convenient answer to almost every uncomfortable AI training question. Need more reasoning traces? Generate them. Need domain examples? Generate them. Need privacy-preserving replacements for customer data? Generate them. Need a dataset that looks suspiciously like a benchmark but not too suspiciously like a benchmark? Generate it, then call it “curriculum design.” ...

GPU memory is the part of AI strategy that does not care about adjectives. A team can say it is building a domain LLM, a private copilot, a long-context research assistant, or a fine-tuned enterprise model. The budget spreadsheet eventually asks a colder question: what actually fits on the available hardware? Model weights need memory. Gradients need memory. Activations need memory. Checkpoints need memory. And the optimizer — the quiet machinery that decides how parameters move during training — can require multiple additional copies of the model itself. ...

Budgets have a charming way of making AI strategy less philosophical. In the demo room, the question is usually whether a model can reason, code, summarize, plan, and sound pleasantly harmless while doing so. In the finance room, the question becomes simpler: how many tokens, how many GPUs, how many weeks, and why exactly are we paying to teach the model another version of the same web page? ...

Training a reasoning model is often treated like running a classroom with a very impatient teacher: give the model a problem, let it produce several answers, mark each answer right or wrong, and push the policy toward the winners. That is already useful. It is also slightly wasteful. Because in a real classroom, the wrong answers are not just trash to be swept off the floor. They reveal what the student misunderstood. They show which shortcuts are tempting, which algebra step keeps breaking, and which false pattern looks suspiciously persuasive. A good teacher does not only praise the correct solution. A good teacher puts the correct and incorrect attempts side by side and asks: what exactly changed? ...

The agent did exactly what it was taught. That was the problem. A familiar business agent failure does not look dramatic. It looks boring. The agent searches the database, clicks the wrong record, receives an error, retries the same action, receives the same error, retries again, and then politely informs the user that it has encountered “temporary difficulty.” Very professional. Completely useless. ...