Choose the Bottleneck Before the KV Cache Strategy

TL;DR for operators

Mamo, Kogiou, Yi, and Yu compare three representative approaches to managing the memory accumulated during LLM generation: keep the full cache on the GPU, permanently discard selected cached tokens, or retain the larger cache in CPU memory and fetch selected entries during decoding.¹ Their benchmark finds no dominant strategy.

For a serving team, the choice depends on what is scarce. vLLM delivers the lowest latency and strongest sustained throughput when GPU memory is sufficient. H2O uses roughly half vLLM’s GPU memory in batch scaling and achieves about three times its throughput per GB of GPU memory, but permanent eviction materially weakens early-context retention. InfiniGen preserves retention close to the full-cache baseline, yet shifts pressure to CPU RAM and repeated host-device transfers; at batch 96 its end-to-end latency reaches roughly 14 times vLLM’s.

The practical rule is to diagnose the bottleneck first. A GPU-capacity problem, a latency problem, a throughput problem, and a contextual-retention problem do not point to the same KV-cache strategy.

Saving GPU memory can create a different serving problem

Long-context serving forces an infrastructure team to allocate a finite hardware budget across competing requirements. More concurrent requests consume more memory. Longer prompts and outputs expand retained state. Moving that state away from the GPU can free accelerator capacity, but the saved bytes may reappear as transfer time, host-memory demand, or information loss.

That retained state is the KV cache: previously computed attention information stored so the model does not recompute the entire preceding sequence for every generated token. Its storage requirement grows with model depth, KV-head count, retained sequence length, head dimension, and numerical precision:

The benchmark compares three ways of handling that growth on a four-H100 node. vLLM represents GPU-resident full-cache management. H2O represents static sparsification, permanently evicting cached tokens according to attention-based importance. InfiniGen represents hierarchical dynamic selection, retaining the larger cache in CPU memory and retrieving selected entries during decoding.

The measurements matter because each approach relocates cost differently.

H2O buys exceptional GPU efficiency with irreversible eviction

The batch-scaling results are the clearest case for static sparsification. vLLM maintains an approximately 72 GB pre-allocated GPU footprint across tested batch sizes. H2O remains below 40 GB even at batch 96 while reaching similar peak throughput. Measured as throughput per GB of GPU memory, H2O reaches approximately three times vLLM’s efficiency.

For a serving operator whose binding constraint is accelerator capacity, that is a material advantage. More requests can fit within the same GPU-memory envelope without introducing the host-device transfer path that slows hierarchical offloading.

The cost appears in what H2O cannot recover. Once a token is evicted, later decoding cannot restore it.

The paper’s sparsity-budget sweep is best read as a sensitivity test for this trade-off. Around a 0.3 retained-KV budget or higher, both sparsification approaches stay within roughly 10 percentage points of vLLM on most of six evaluated reasoning datasets. At extreme 2–4% budgets, performance on semantic reasoning tasks approaches random guessing. At a 0.1 budget, H2O records losses including 23 percentage points on HellaSwag and 16 points on BoolQ.

The dedicated retention experiment probes the failure mode more directly. Researchers inserted a verifiable fact near the beginning of multi-turn conversations and queried it again at later turns. At a 0.3 budget, vLLM retains 94–96% accuracy and InfiniGen 92–94%. H2O reaches only 70–78%, with its deficit increasing across longer contexts.

For applications that depend on retrieving earlier contractual terms, user instructions, case facts, or other long-lived conversational state, GPU-memory efficiency therefore has to be priced together with the cost of unrecoverable eviction. The paper does not establish how severe that cost will be for every application; its retention test measures injected factual recall, not the full range of long-context reasoning.

InfiniGen protects retention by taxing the decode path

Hierarchical offloading addresses H2O’s central weakness differently. Instead of permanently deleting most cached entries, InfiniGen keeps the larger cache in CPU memory and transfers selected entries back to the GPU when needed.

The retention results show why that architecture is attractive: its injected-fact accuracy stays close to vLLM and well above H2O.

Operationally, however, the cache has not disappeared. It has moved.

At batch 96, InfiniGen’s CPU-memory consumption exceeds 100 GB, more than 2.5 times that of the alternatives. More consequentially for serving speed, CPU-to-GPU KV transfers recur at every generation step. The work cannot simply be amortized across an entire request.

The resulting decode penalty is large. vLLM and H2O scale throughput roughly linearly with batch size and reach similar peak rates, while InfiniGen remains about an order of magnitude slower. At batch 96 its end-to-end latency reaches approximately 14 times vLLM’s. Across output lengths from 512 to 8,192 tokens, vLLM sustains about 15 times InfiniGen’s throughput. At 8,192 generated tokens, the measured completion times are roughly one minute for vLLM, three minutes for H2O, and more than 16 minutes for InfiniGen.

For a latency-sensitive interactive product, that transfer path is not a secondary implementation detail. It can become the principal serving bottleneck.

Long-context failures can happen before KV capacity is exhausted

The study also isolates a separate failure mode that capacity planning can miss.

Baseline H2O and InfiniGen encounter out-of-memory failures around 10K input tokens even though the evaluated models support much longer contexts. The limiting allocation is not initially the retained KV cache. Their prefill paths materialize full attention-score matrices, whose memory requirements become prohibitive first.

The paper’s FlashAttention-2 and chunked-prefill experiments function as bottleneck-mitigation tests. For InfiniGen, FlashAttention-2 reduces prefill memory by 85% and latency by 77% relative to chunked prefill at 15K tokens and enables operation out to 128K tokens.

H2O behaves differently because Heavy-Hitter selection still requires the full attention-score matrix. Chunked prefill can extend its feasible context length, but each chunk sees incomplete cross-chunk attention information when making eviction decisions.

The infrastructure lesson is broader than nominal cache size. Long-context capacity planning should separately account for KV storage, prefill working memory, CPU RAM, and host-device bandwidth. Optimizing one line item does not guarantee that it is the line item currently limiting the system.

Match the strategy to the binding constraint

Cognaptus interprets the measurements as a constraint-selection problem for serving architecture.

A team operating latency-sensitive APIs or high-throughput generation should favor GPU-resident cache management when sufficient accelerator memory is available. The tested vLLM configuration avoids both permanent eviction and recurrent CPU-GPU transfer costs.

A team primarily blocked by GPU capacity can consider static sparsification when the application can tolerate some loss of earlier context. H2O’s measured GPU-memory efficiency is substantial, but the acceptable sparsity budget should be validated against application-specific quality tests rather than maximized mechanically.

A memory-constrained system that values recoverable context more highly than request throughput has a different option in hierarchical offloading. InfiniGen demonstrates that recoverability can protect retention, but on the tested hardware it converts GPU pressure into CPU capacity requirements and serialized data movement.

Hardware changes can alter that last calculation. Faster host-device interconnects could reduce the economic penalty of hierarchical placement, which means a serving decision made on one generation of infrastructure should not automatically survive the next.

The benchmark defines operating points, not universal rankings

The study provides controlled comparative evidence, but its deployment boundaries matter.

Only one representative framework is tested for each broad paradigm. H2O is implemented by the authors on FlexGen rather than evaluated as an unchanged original serving stack, while H2O and InfiniGen use sequential prefill followed by batched decoding. All systems measurements come from one four-H100, Sapphire Rapids environment with NVLink and PCIe 5.0. The principal system comparisons also concentrate around a 0.3 sparsity budget.

Those conditions prevent the results from proving that every static-sparsification or hierarchical-offloading system will reproduce the same numbers.

They do support a more durable operating rule. KV-cache optimization does not eliminate serving cost; it redistributes cost among accelerator capacity, computation, data movement, and retained information. The correct deployment choice begins by identifying which of those constraints is actually limiting the workload.

Cognaptus: Automate the Present, Incubate the Future.