Llmops

TL;DR for operators RAG teams often want to believe that a smarter chunking method will rescue messy document retrieval. It is a tidy belief. It is also the sort of tidy belief that tends to become a budget line. The paper behind this article tests that belief in a small, practical setting: thirteen academic theses, ten questions per thesis, three chunking strategies, and a self-hosted RAG stack constrained by 16 GiB of VRAM.1 The strategies are familiar: fixed-size chunks, recursive format-aware chunks, and cluster-based semantic chunks. The expensive-sounding one, cluster-based semantic chunking, does not consistently win. ...

Enterprise AI teams have developed a familiar reflex. When the model behaves unreliably, they try a better prompt. When that fails, they try a larger model. When that becomes expensive, they invent a workflow diagram with many arrows and call it an operating model. Very dignified. Very scalable, in the same way that adding more sticky notes to a broken process is scalable. ...

A support ticket does not usually fail because every message was bad. More often, it fails because one reply arrived at exactly the wrong moment: the bot misunderstood a frustrated customer, repeated a stale answer, missed the escalation point, and then ended the interaction with something sterile enough to pass a benchmark but useless enough to make the customer leave. The average quality may look acceptable. The experience still feels broken. ...

Shipping Is the Part Agents Usually Skip Shipping is where confidence goes to die. A demo agent can impress everyone on Tuesday, receive a clever prompt update on Wednesday, and quietly break three workflows that were working last week. The aggregate score improves. The release notes look cheerful. Somewhere, a previously solved customer task becomes unsolved again. Naturally, everyone calls this “iteration,” because “we broke production while chasing a benchmark bump” sounds less strategic. ...

TL;DR for operators LLM application testing should stop pretending that the whole product behaves like ordinary software. The database connector, retry logic, API wrapper, and schema validator still deserve normal unit, integration, and load tests. Fine. Keep those. They are not the problem. The problem starts when the product becomes a stateful language system: prompts are assembled dynamically, retrieval changes the context, tool calls modify the execution path, memory leaks across turns, and a model update can improve one workflow while quietly breaking another. At that point, exact-match assertions become less like QA and more like theatre with a YAML file. ...

TL;DR for operators A model upgrade is not a software patch. It is closer to changing the interpreter under a production system while hoping every old script still means the same thing. Charming, in the way live wires are charming. The paper behind this article, Prompt Migration: Stabilizing GenAI Applications with Evolving Large Language Models, studies that problem through Tursio, an enterprise search application that converts natural-language questions into structured operator trees for database querying.1 Tursio’s old prompts were fully stable on GPT-4-32k. When the same prompts were run against GPT-4.1, tests passed at 98%. Against GPT-4.5-preview, they passed at 97.3%. That sounds minor until the application is generating SQL-like structures, where “almost correct” is not a governance model. ...

TL;DR for operators A network outage is not a single question. It is a sequence: probe reachability, inspect counters, compare paths, refine the hypothesis, ask for better telemetry, and decide whether to act. That sequence is exactly where static LLM benchmarks become rather ornamental. A model that can answer a configuration question offline is not necessarily an agent that can diagnose a live fault while the network keeps misbehaving. ...