Where to Go Deeper Beyond This Academy

A curated guide to textbooks, authors, websites, and papers for readers who want to study transformer internals, attention math, fine-tuning, GPU optimization, and benchmarking in more depth.

April 23, 2026 · 9 min · Michelle
Cover image

When 8,000 Examples Beat 80,000: What a Model Still Needs to Learn

TL;DR for operators A team has 80,000 instruction-response examples but enough resources to fine-tune on only 8,000. Selecting the examples with the highest model loss may prioritize genuinely missing knowledge, but it can also prioritize noise. Random selection and conventional quality ratings do not necessarily reveal what the current model can already retrieve. ...

October 30, 2026 · 10 min · Zelina
Cover image

The Vanishing Quotient: When More Training Erases Arithmetic Skills

TL;DR for operators A compact AI model can learn a numerical procedure, perform it successfully in testing, and then lose that capability during further training. Sourabh Kasliwal’s study of tiny Transformers1 illustrates this sharply: a model correctly solved 86.67% of 300 division problems with single-digit divisors, but its accuracy on that same subset fell to 0% after another 20,000 mathematical training steps. ...

October 29, 2026 · 9 min · Zelina
Cover image

Where You Pause Changes What You Forget

TL;DR for operators Kim and colleagues’ study of pause-token fine-tuning1 points to a training rule that is easy to miss if pause tokens are treated mainly as extra thinking time. At an equal pause-token budget, putting pauses at semantic boundaries is the only tested placement that consistently improves both math and code averages over ordinary supervised fine-tuning. The stronger variant also masks the loss on the pause tokens themselves. ...

October 1, 2026 · 6 min · Zelina
Cover image

When the Proxy Wins: Attention Sensitivity Can Hit the Ceiling and Still Miss ICL

TL;DR for operators An internal metric can track a capability well enough to be useful for diagnosis and still fail once the training loop is told to maximize it. In this study, an attention-based signal intended to reflect whether demonstrations matter to a model rises from about 0.516 to 1.413 under direct optimization—within 0.5% of its theoretical maximum. Yet the corresponding behavioural effect is essentially absent on the controlled QA probe, and MMLU accuracy falls to 0.279.1 ...

September 23, 2026 · 7 min · Zelina
Cover image

The Fourth Hop Changes the Risk Profile: Measuring Reliability in Multi-Step LLM Workflows

TL;DR for operators When one model output becomes input to the next stage, a final accuracy score tells you too little about where reliability is being lost. A workflow may fail because a required fact was never available, because a later composition step is intrinsically harder, or because an earlier mistake was allowed to propagate. Those failure modes call for different controls. ...

September 17, 2026 · 8 min · Zelina
Cover image

Fine-Tuning Changes What Your Model’s Errors Reveal

TL;DR for operators A fine-tuned model can become only slightly more accurate while its remaining errors become substantially easier to distinguish from correct answers. That matters when uncertainty scores feed operational controls. If a production workflow accepts an answer, abstains, calls another model, or sends a case to human review according to a detector threshold, fine-tuning changes more than the benchmark score. It can change the detector itself as an operating signal. ...

September 10, 2026 · 7 min · Zelina
Cover image

Compress the Activation, Spend the Memory on Rank

TL;DR for operators A LoRA fine-tuning job can fit its trainable parameters comfortably on a GPU and still run out of memory because backpropagation retains large intermediate activations. CARE-LoRA attacks that remaining buffer rather than shrinking the adapter itself. Zhang et al. store LoRA’s already-compressed activation plus a small reconstruction matrix, then use those tensors to approximate only the gradient needed for one of LoRA’s two trainable matrices.1 ...

September 5, 2026 · 7 min · Zelina
Cover image

LoRA’s Missing Budget: Which Matrices Deserve an Adapter?

TL;DR for operators LoRA already avoids the cost of updating an entire pretrained model, but it can still spend adapter capacity uniformly across matrices that do not appear equally responsive to low-rank changes. If a training team has a fixed fine-tuning budget, there are therefore two allocation decisions: how large each adapter should be, and which matrices should receive one. ...

September 5, 2026 · 7 min · Zelina
Cover image

Train Wide, Deploy Narrow: LoRA Rank Does Not Have to Be One Decision

TL;DR for operators A production team may want every LoRA adapter to fit a small, uniform serving footprint. The usual response is to choose that small rank before training and optimize inside the resulting constraint. This paper shows that the training capacity and the deployment capacity do not always need to be identical. ...

September 5, 2026 · 7 min · Zelina