LLM Operations

Long reasoning has become the new luxury trim in AI products. The demo no longer just answers. It pauses, reflects, reconsiders, checks itself, writes a small philosophical memoir, and then hopefully solves the problem. This is not entirely theatrical. Chain-of-thought style reasoning and large reasoning models have improved performance on difficult tasks, especially in mathematics, coding, planning, and multi-step analysis. For business users, that matters. A model that can break down a problem is more useful than one that confidently blurts out the first plausible answer. Nobody wants a legal assistant, financial analyst, or production-support agent whose main cognitive strategy is “vibes, but fast.” ...

Opening — Why this matters now AI reasoning has entered its awkward managerial phase. For the past two years, the dominant story has been simple enough for a conference keynote: make models reason longer, use reinforcement learning, scale inference-time computation, and let the model “think.” The story is not wrong. It is just incomplete in the same way that saying “hire more analysts” is an incomplete operating model for a research department. More thinking can help. It can also become expensive, slow, noisy, and occasionally theatrical. ...

A car does not care whether an AI answer is impressive. It cares whether the answer arrives before the intersection. That small timing problem is where a large part of today’s agentic AI discussion becomes unserious. We keep asking whether models are smart enough to act. In cyber-physical systems, the more painful question is whether the system around the model can make action repeatable, bounded, and recoverable when the model is late, vague, or simply wrong. ...

Models pile up quietly. A customer-support model here. A finance QA model there. A legal drafting variant that nobody wants to delete because it passed last quarter’s evaluation. A sales assistant fine-tuned on a dataset that may or may not still represent how the company sells. Then come LoRA adapters, instruction-tuned checkpoints, safety-tuned variants, regional versions, and a few “temporary” experiments that become permanent because nobody enjoys breaking production on a Friday. ...

Idle time is not empty time. Anyone who has managed a human team already knows this. Leave a capable person with no clear assignment and they may tidy the backlog, invent a side project, interrogate the process, or spend the afternoon constructing a philosophy of why the calendar is oppressive. Large language model agents, apparently, have their own version of this behaviour. Less caffeine, more JSON, same managerial problem. ...

TL;DR for operators Most agent teams eventually discover that “the workflow” is not one thing. A customer-support agent, a coding agent, and a mathematical reasoning agent may all use decomposition, verification, consensus, and answer extraction—but not in the same order, not with the same emphasis, and definitely not with the same failure modes. Static agent templates look tidy in architecture diagrams. Then the first heterogeneous workload arrives, and the diagram starts quietly sweating. ...

TL;DR for operators A user asks an AI agent to delete an account. The obvious tool is DeleteAccount. A normal semantic retriever will probably find it. Splendid. The agent still fails if it misses GetUserToken, because the deletion tool needs a token first. This is the failure mode Tool Graph Retriever, or TGR, is built to address.1 ...

TL;DR for operators Adapters look small. The privacy surface is not. The paper behind LoRA-Leak argues that LoRA fine-tuning does not magically protect the records used to specialise a language model.1 Even though LoRA trains only low-rank adapter weights while leaving the base model frozen, the resulting model can still leak membership information: an attacker may infer whether a given sample was part of the fine-tuning dataset. ...

TL;DR for operators A recent case study from If P&C Insurance is useful because it does something most AI automation stories conveniently skip: it follows the work after the model is deployed.1 The company used an LLM to identify specialised claim parts in insurance claims, a task that had depended on human claim handlers and specialist knowledge. In offline evaluation, the fifth model iteration built around GPT-4o-0806 reached 81% recall in English, above the company’s 70% human baseline. That sounds like the usual “AI beats humans” headline. Mercifully, the paper is more interesting than that. ...

TL;DR for operators AI reasoning is becoming an operating cost, not just a research curiosity. When a model “thinks step by step,” every intermediate token has to be generated, paid for, waited on, logged, and sometimes hidden from the user because nobody wants a customer support bot narrating its algebra like a nervous intern. ...