Waiting is the least glamorous part of an AI agent. A user asks for a report, a code fix, a dataset analysis, or a literature scan. The agent thinks, calls a tool, waits, reads the result, thinks again, calls another tool, waits again, and repeats this little ritual until the final answer appears. From the outside, this looks like “reasoning.” From the system side, much of it is simply queueing around tools. ...
A deployment team has a large model, a smaller device, and a familiar problem: the model is too heavy for the place where the business actually wants to use it. So the team reaches for the standard efficiency drawer. Prune some weights. Quantize the remaining values. Maybe add a light adapter to recover accuracy. Push the result to edge hardware, a mobile app, or a cheaper inference server. Then explain to management why the model became faster but also slightly less intelligent. The usual ritual. ...
Routing is the unglamorous part of agentic AI. Which is exactly why it matters. A company can assemble a neat little digital workforce: one agent plans, one agent searches, one agent codes, one agent critiques, one agent writes the final answer. It looks sophisticated on a diagram. Then production traffic arrives, and the system discovers a more ancient truth: a committee is not useful if every request goes through the wrong people in the wrong order. ...
AI has a landlord problem. Not because models are renting office space, although given GPU bills, perhaps they should negotiate. The deeper issue is that modern AI increasingly lives inside a small number of large platforms. The data, the compute, the model weights, the deployment channels, the safety policies, and often the user interface are controlled by the same narrow set of institutions. The result is not merely concentration in a business-school chart. It is concentration in the machinery through which other businesses now write, decide, recommend, price, design, and automate. ...
A demo can survive a slow answer. A production service cannot survive the slow answer that arrives just often enough to make users stop trusting the product. That is the quiet problem behind p99 latency. The average response time tells you how the service feels on a normal day. p99 tells you what happens to the unlucky one percent: the support agent waiting in front of a customer, the analyst refreshing a dashboard, the employee whose workflow now includes watching a spinner and reconsidering their life choices. ...
Traffic lights are not stupid. They are obedient. That is the problem. A fixed-time signal does exactly what it was told to do: hold this green for this long, clear the junction, move to the next phase, repeat. It does not care that one lane is empty, another is spilling backward, and a third has just received a platoon of vehicles from the previous intersection. It is not being malicious. It is merely following a plan designed for a world that stopped changing five minutes ago. ...
Waiting is the least glamorous part of AI. A user uploads a contract, a codebase, a board pack, or a pile of research notes. The model does not answer immediately. First, it reads. Technically, it prefills: it processes the prompt, builds the internal key-value cache, and prepares the first generated token. In short prompts this feels invisible. In long-context systems, it becomes the awkward pause where the “agent” looks suspiciously like a very expensive loading spinner. ...
Numbers look harmless until they enter a business database. A revenue field says 50. A dosage field says 50. An age field says 50. A follow-up period says 50. A unit may be present, missing, abbreviated, buried in the column header, or inconsistently written as ml, mL, or something the spreadsheet inherited from a PDF extraction pipeline during its villain era. ...
Serving an LLM is usually discussed in pleasantly managerial language: latency, throughput, context windows, GPU memory, quantization, cache eviction. Nice clean nouns. Then the model ruins the spreadsheet by producing internal activations that are thousands of times larger than ordinary values, while some tokens quietly become attention magnets for reasons that are not exactly semantic. Very professional behavior from a trillion-dollar technology stack. ...
Screen. That is where many ambitious AI agents quietly embarrass themselves. Not in a grand philosophical test of intelligence. Not in a graduate-level theorem. Just on a screen: a small button, a chart label, a checkout field, a misread table cell, a tiny icon in a crowded interface. The model can explain strategy, summarize policy, and generate six polite versions of an apology email, but then it clicks the wrong thing because it did not really see the thing. ...