Long-Context

TL;DR for operators Long context becomes expensive for a specific architectural reason: standard self-attention allows every position to interact with every other position. In the usual formulation, that means $O(n^2 d)$ computation and $O(n^2)$ storage for attention-related matrices as sequence length $n$ grows. Hasi Hays’s mathematical monograph on attention1 is useful because it connects that deployment constraint directly to the mechanism that makes attention powerful in the first place. ...

TL;DR for operators Long-context prefill creates an uncomfortable infrastructure choice. Lowering attention precision can reduce compute cost, but applying INT8 everywhere may degrade retrieval and question-answering quality. Sparse attention attacks the same cost problem differently by avoiding some interactions altogether. TileMix proposes another control surface: keep every legal attention interaction, but choose which regions of the attention-score computation run in FP16 and which run in INT8. The paper1 reports that this mixed routing can recover part of the quality lost under uniform INT8 while retaining much of the throughput advantage. On an NVIDIA A100 40GB, LLaMA 3.2 3B-Instruct at 4k tokens reaches 31.80 K tokens/s with the SpTrans75 configuration, versus 14.33 K tokens/s for FlashAttention and 29.80 K tokens/s for the matched all-INT8 path. ...

TL;DR for operators Long-context teams face a familiar choice: compute fewer token-to-token interactions, compress global interaction structure, or accept the quadratic cost of exact attention. The harder design question is what happens when two cheaper approximations are combined. If each branch normalizes its own output over a different effective support, simply adding or gating them can give the branches incompatible scales. ...

TL;DR for operators When an AI system reads a manual, policy, API specification, case file, or operating procedure, finding the relevant facts is only half the job. It must also discover the local rules that define what a valid answer looks like: required fields, exact labels, ordering constraints, exception handling, validation steps, prohibited actions, and completeness conditions. ...

A codebase is not a paragraph. Neither is a litigation folder, a clinical case file, a customer-support history, a policy archive, or the slow-motion disaster known as “all meeting notes since March.” Yet many enterprise AI systems still treat long context as a heroic prompt-engineering problem: push more text into the model, pray the key detail survives attention, and call the bill “innovation.” ...

Waiting is the least glamorous part of AI. A user uploads a contract, a codebase, a board pack, or a pile of research notes. The model does not answer immediately. First, it reads. Technically, it prefills: it processes the prompt, builds the internal key-value cache, and prepares the first generated token. In short prompts this feels invisible. In long-context systems, it becomes the awkward pause where the “agent” looks suspiciously like a very expensive loading spinner. ...

A customer tells your AI assistant that she dislikes crowded tourist attractions. Three weeks later, she asks for a weekend itinerary. A good assistant should not proudly recommend the busiest landmark in the city. A less good assistant will do exactly that, but in a warm tone. This is the quiet failure mode behind many “personal AI” demos. The interface remembers the conversation. The product claims continuity. The model may even have a giant context window large enough to swallow a small novel. Yet when the user asks a new question, the system behaves as if the earlier preference is just decorative text floating somewhere in the attic. ...

Documents are the easiest way to fool an AI system into looking serious. A procurement team uploads the full contract archive. A compliance team adds policy manuals, audit notes, and emails. A financial analyst stuffs transcripts, filings, and market commentary into one heroic prompt. The interface accepts it. The model answers fluently. Everyone relaxes. ...

A support agent keeps asking the same diagnostic question after the customer has already answered it. A research agent revisits the same failed source path with slightly different wording. A workflow agent tries the same invalid action again because, apparently, the best evidence for what to do next is what it just did badly. ...

Laptop. That is the deceptively simple object hiding inside this paper. Not a magic planner. Not a thousand-tool agent marketplace. Not a baroque workflow with seventeen orchestration layers and a dashboard that looks like a cockpit designed by consultants. A laptop. Or, more precisely, a minimal virtual computer: a sandbox with terminal access, file editing, code execution, persistent files, and the ability to install or fetch resources. In Computer Environments Elicit General Agentic Intelligence in LLMs, Cheng et al. ask a question that looks almost too obvious to be interesting until one remembers how much of the AI industry is still trying to squeeze “agency” out of longer prompts.1 ...