Model-Evaluation

TL;DR for operators A useful AI education product does not always need the largest model in the room. Sometimes it needs a smaller model that has been taught one job properly and then told, firmly, not to hand students the answer on a silver platter. The paper behind this article studies exactly that: whether supervised fine-tuning can make open-source models good enough to explain C programming errors for novice students. The authors use real CS1/2 error logs from DCC Help, generate 40,000 structured explanations with GPT-4.1, fine-tune Qwen3-4B, Llama-3.1-8B, and Qwen3-32B using QLoRA, then compare them against base models, GPT-4.1, and the original deployed DCC Help responses. ...

TL;DR for operators When an LLM leaks sensitive, copyrighted, or otherwise forbidden information, the obvious repair is to fine-tune it away from the bad answer. That sounds sensible until you notice the small operational comedy: the remediation process keeps using the very answer it is supposed to remove. The paper behind this article proposes Partial Model Collapse (PMC), a machine unlearning method that avoids directly optimising on ground-truth forget answers. Instead, PMC asks the model the sensitive question, samples multiple responses from the model itself, selects a response that is less like the model’s original answer, and fine-tunes on that self-generated response while also training on retain data to preserve general utility.1 ...

TL;DR for operators Text does not automatically make forecasts smarter. It often just makes the pipeline heavier. A new AWS study benchmarks multimodal time-series forecasting across 16 datasets and 7 domains, comparing time-series-only models, alignment-based multimodal models, and direct LLM prompting.1 The uncomfortable result is that multimodality is not a universal upgrade. Strong unimodal models still win on a substantial share of the benchmark, and the paper’s statistical tests do not support a blanket claim that adding text reliably improves accuracy. ...

TL;DR for operators Real business problems do not arrive as tidy exam questions. They arrive as “Can we optimise this logistics network?”, “Which markets should we prioritise?”, “How many clinics do we need?”, or “What happens if the subsidy disappears?” The annoying part is not the equation. The annoying part is deciding what the equation should even represent. ...