Cover image

Beyond Stack Overflow: CodeAssistBench Exposes the Real Gaps in LLM Coding Help

TL;DR for operators Coding assistants look much better when the task is a clean question than when the task is a messy software support conversation. That is the inconvenient point of CodeAssistBench, or CAB, a benchmark that turns resolved GitHub issues into multi-turn, project-grounded conversations where a model must behave like a maintainer, not a code-snippet vending machine.1 ...

July 16, 2025 · 17 min · Zelina
Cover image

From Prompting to Porting: Surviving the LLM Upgrade Cycle

TL;DR for operators A model upgrade is not a software patch. It is closer to changing the interpreter under a production system while hoping every old script still means the same thing. Charming, in the way live wires are charming. The paper behind this article, Prompt Migration: Stabilizing GenAI Applications with Evolving Large Language Models, studies that problem through Tursio, an enterprise search application that converts natural-language questions into structured operator trees for database querying.1 Tursio’s old prompts were fully stable on GPT-4-32k. When the same prompts were run against GPT-4.1, tests passed at 98%. Against GPT-4.5-preview, they passed at 97.3%. That sounds minor until the application is generating SQL-like structures, where “almost correct” is not a governance model. ...

July 9, 2025 · 18 min · Zelina
Cover image

What Happens in Backtests… Misleads in Live Trades

TL;DR for operators A beautiful backtest can still be a lie. Not because the model is malicious, obviously; spreadsheets have not yet formed a union. The problem is simpler and more expensive: a model can fit past data while misrepresenting the thing you actually care about. Charles Rathkopf’s paper on hallucination and reliability in scientific generative AI gives operators a useful way to think about this problem.1 It argues that hallucination should not be defined mainly as deviation from training data. In science, and in business domains that behave like science, the real question is whether an output misrepresents the target phenomenon: a protein, a weather system, a molecule, a patient, a market, a factory, a supply chain. ...

April 15, 2025 · 17 min · Zelina