Precision Has a Map: TileMix Routes INT8 Inside Dense Attention

TL;DR for operators

Long-context prefill creates an uncomfortable infrastructure choice. Lowering attention precision can reduce compute cost, but applying INT8 everywhere may degrade retrieval and question-answering quality. Sparse attention attacks the same cost problem differently by avoiding some interactions altogether.

TileMix proposes another control surface: keep every legal attention interaction, but choose which regions of the attention-score computation run in FP16 and which run in INT8. The paper¹ reports that this mixed routing can recover part of the quality lost under uniform INT8 while retaining much of the throughput advantage. On an NVIDIA A100 40GB, LLaMA 3.2 3B-Instruct at 4k tokens reaches 31.80 K tokens/s with the SpTrans75 configuration, versus 14.33 K tokens/s for FlashAttention and 29.80 K tokens/s for the matched all-INT8 path.

For an LLM-serving team, the relevant decision is therefore not simply whether to quantize attention. It is how much of the score computation to route to INT8, and where those lower-precision regions should sit. That policy requires workload-specific benchmarking rather than a fixed universal setting.

Dense attention does not require uniform precision

The usual acceleration choices couple two decisions that do not have to be coupled.

Uniform quantization says that if attention is going to run at lower precision, the same arithmetic treatment should apply broadly across the operator. Sparse attention instead reduces work by removing selected token-to-token interactions. TileMix separates connectivity from precision.

Every legal interaction remains part of the dense attention computation. What changes is the numerical format used for the hardware-aligned score tiles containing those interactions. Some tile groups use FP16 QK score computation; others use blockwise INT8 QK computation with INT32 accumulation and rescaling back into the common floating-point score domain.

That distinction matters operationally. A serving team can seek lower prefill cost without changing the model weights and without redefining which tokens may attend to which others. Precision becomes spatially allocatable inside the existing dense-attention computation.

The paper calls this tile-group precision routing. Its coverage settings—25%, 50%, or 75%, for example—refer to the share of legal score-tile groups routed to INT8. They do not mean that 25%, 50%, or 75% of attention interactions have been removed, nor do they directly state the fraction of total attention FLOPs executed in INT8.

The routing mechanism is designed to stay inside the hot loop

Mixed precision would be less compelling if selecting a path required expensive control logic or separate normalization machinery.

TileMix addresses the control problem by packing the routing decisions for a KV-head/query-tile-row pair into a 64-bit word. The kernel retrieves each decision with a shift-and-mask operation. Adjacent key tiles can be grouped under one bit so the number of routing decisions remains at most 64 even as sequence length grows.

The more consequential design choice is what happens after the two arithmetic paths produce scores. INT8 results are rescaled into the same floating-point domain as FP16 results, and both update one common online-softmax maximum, normalizer, and output accumulator. The mixed paths therefore participate in one attention normalization process rather than behaving like two separately computed attentions stitched together afterward.

This is what makes the architecture relevant as a systems contribution rather than only a precision-selection idea. The routing decision is compact enough to sit inside fused streaming attention, and heterogeneous arithmetic does not require abandoning the shared state used by FlashAttention-style execution.

The performance result is strong, but coverage alone does not predict quality

The paper’s main implementation evidence is its A100 throughput comparison. At 4k tokens and batch size 8, the reported LLaMA 3.2 3B-Instruct results are:

The 31.80 K tokens/s result is about 2.2× the reported FlashAttention throughput under this protocol. More interestingly, the mixed configurations operate in roughly the same throughput range as the matched all-INT8 implementation while the long-context evaluations generally show better retrieval and QA quality than uniform INT8.

That does not mean 75% INT8 coverage is automatically the preferred configuration. The quality tests across LongEval and LV-Eval show that routing layout matters at a given nominal coverage. BigBird-, Row-Random-, and Sparse-Transformer-inspired patterns are among the more consistent layouts, but the best configuration varies across model family, task, context length, and coverage.

The numerical-deviation experiments serve a different purpose from the task benchmarks: they are sensitivity tests for the precision control itself. Mean absolute deviation from the FP16 reference generally rises as more tile groups are assigned to INT8. At 4k tokens, for example, reported mean deviation increases from $4.87\times10^{-5}$ at 0% INT8 routing to $1.78\times10^{-3}$ at 25%.

That makes coverage a measurable numerical control knob, but not a sufficient policy by itself.

Placement can matter more than the nominal percentage

An appendix analysis helps explain why two layouts with the same INT8 coverage need not behave alike.

For the static SpTrans25 configuration, only about 8.5% of the selected high-importance attention mass falls onto INT8-routed regions even though 25% of legal tile groups are assigned to INT8. This is best read as a mechanism probe, not as proof that static Sparse-Transformer-style routing will identify important interactions for every model or input.

Still, it shows the mechanism the design is exploiting: spatial structure can make the location of reduced precision more consequential than the headline coverage ratio. A static routing template may preserve disproportionately more consequential score regions in FP16 without running an online importance detector for each request.

For infrastructure teams, that changes the tuning problem. “Use 50% INT8” is underspecified. The deployment policy is closer to “use this coverage under this spatial layout for this model and workload.”

Cognaptus interpretation: precision routing becomes a serving policy

The paper directly establishes an A100 implementation that can trade score precision spatially inside dense attention, with measurable effects on throughput, benchmark quality, and numerical deviation.

Cognaptus infers a broader infrastructure pattern: numerical precision can be treated as a schedulable resource rather than a single operator-level setting. For teams serving long documents, retrieval-heavy prompts, or long-context QA, that could support model-specific profiles such as a conservative routing layout for accuracy-sensitive traffic and a higher-INT8 configuration when prefill latency dominates the service objective.

The relevant user is the inference-platform team. The decision is the routing layout and INT8 tile-group coverage applied during prefill. The condition is a workload where dense long-context attention materially contributes to latency or GPU time. The measurable boundary is the resulting task-quality and numerical-deviation tolerance on the target model and hardware.

No retraining is required for the evaluated static policies, which lowers the operational cost of experimentation. But this should be treated as kernel tuning, not as a model-independent optimization setting.

The A100 result should not be exported unchanged to other deployments

The evidence is strongest for forward inference during long-context prefill on NVIDIA A100 GPUs using FP16 and INT8 Tensor Core paths. The primary path keeps V and the PV computation in FP16; it is not an end-to-end INT8 attention implementation. The paper also implements an INT8 key/value-cache interface, but that is separate from the main prefill evaluation.

The routing policies evaluated most extensively are static and data-free. Adaptive or learned routing remains a plausible extension, not an established advantage in this study.

Most importantly, hardware architecture enters directly into the economics. Scale handling, Tensor Core behavior, memory movement, scheduling, and kernel geometry can differ on another accelerator. A serving team should therefore reproduce the latency-quality frontier on its target GPU, model family, context distribution, and task mix before promoting a routing configuration into production.

TileMix’s more durable contribution is the abstraction behind those benchmarks. Long-context attention need not choose between one precision everywhere and fewer interactions overall. Once precision can be assigned spatially inside dense attention, the question becomes which regions deserve the expensive arithmetic—and how cheaply the system can enforce that decision.

Cognaptus: Automate the Present, Incubate the Future.