Shrink the KV Cache, Miss the Bottleneck

TL;DR for operators

You can cut KV-cache memory substantially and still leave end-to-end latency almost unchanged.

That is the central practical message of Jiang et al.’s survey of serving-time KV-cache optimization.¹ The literature does not point to one interchangeable family of “KV optimizations.” Different techniques intervene at different points in the serving system: some change when KV work executes, some change where KV state resides or moves, and some change how much state is represented or retained.

The distinction matters because these behaviors attack different constraints. Scheduling and execution techniques most directly address latency and throughput. Placement and migration primarily address interconnect and memory-tier movement. Compression and retention management dominate direct GPU-memory reduction, but may introduce quality loss or runtime overhead.

For an inference operator, the first question should therefore be: what resource is actually binding? If GPU memory is limiting concurrency, structural changes may help. If devices are idle while data moves, further compression may do little unless transfer and execution are redesigned together. If the problem is tail latency, average memory savings are an incomplete success criterion.

The survey also finds that hardware-aware execution and compute-device orchestration are the most strongly associated pair in the literature, with a normalized co-design affinity of 0.53. Compression, despite receiving substantial research attention, has affinities of only 0.00-0.06 with the other behavior families. That is descriptive evidence about research practice, not proof that one architecture is superior.

A smaller cache does not guarantee a faster system

Consider an inference team that has already reduced KV memory enough to fit more concurrent requests on each GPU. The memory dashboard improves, yet tokens per second barely move and request latency remains stubborn.

Compressing further is one possible response. So is changing request scheduling, overlapping transfers with computation, moving KV state across GPU, CPU, CXL, or storage tiers, or restructuring which devices perform which parts of serving.

These are not equivalent interventions.

The survey frames the problem around serving-time system actions on KV state for off-the-shelf models, excluding approaches that require retraining or changes to the model architecture. It calls this scope system-aware KV infrastructure for serving LLMs, or sKis.

Its taxonomy asks three simple questions:

The taxonomy contains seven more specific categories, including KV-centric scheduling, pipelining and overlapping, hardware-aware execution, memory-hierarchy orchestration, compute-device orchestration, compression, and retention management.

This framing changes the engineering question from “How do we shrink KV?” to “Which system resource are we trying to release?”

Different KV techniques solve different resource problems

The paper’s behavior-objective synthesis shows a clear division of labor across the literature.

Temporal techniques have the strongest direct relationship with mean latency and throughput. Scheduling can reduce stalls; overlapping can hide communication or I/O behind useful work; hardware-aware execution can reduce device under-utilization.

Spatial techniques instead operate on the path KV state takes through the system. Memory-hierarchy orchestration moves KV state among memory tiers, while compute-device orchestration changes where processing occurs. Their main value is not compression itself but controlling expensive data movement and making heterogeneous resources usable.

Structural techniques are where the literature concentrates most heavily. Compression reduces representation size through mechanisms such as lower precision, lower rank, or sparsity. Retention management controls which state remains available through allocation, reuse, eviction, recall, or merging. These methods directly relieve GPU-memory pressure and can increase feasible context length or concurrency.

But structural savings create two additional questions: what work is required to use the smaller representation, and what information has been discarded?

The survey explicitly warns that lower bitwidth or a smaller representation does not guarantee lower latency or higher throughput. Decompression cost, kernel boundaries, data transfers, and poor runtime integration can absorb the local memory gain.

An operator evaluating a compression method therefore needs an end-to-end measurement path:

memory saved → extra runtime work → transfer behavior → scheduling interaction → realized latency and throughput.

Stopping at the first term can make a successful component optimization look like a successful serving optimization when it is not.

The co-design map points toward execution and movement

The paper goes beyond categorization by measuring how often different behaviors appear together in surveyed systems.

It weights a paper’s primary category as 1 and a secondary category as 0.5, computes weighted co-occurrence, and normalizes the result using a Tanimoto coefficient. The resulting score is a literature co-design affinity: a measure of shared presence after accounting for how common each behavior is individually.

The strongest observed pair is hardware-aware execution with compute-device orchestration at 0.53. Other relatively strong connections include hardware-aware execution with overlapping at 0.42, overlapping with compute-device orchestration at 0.38, and overlapping with memory-hierarchy orchestration at 0.30.

Compression looks very different. Its affinity with every other category is only 0.00-0.06.

The useful interpretation is not that compression performs poorly. The survey does not test that proposition. Rather, a heavily studied method for reducing KV size is comparatively disconnected from the execution and placement mechanisms that determine whether those savings survive into system-level performance.

For serving architects, that makes compression-plus-runtime integration a promising design hypothesis. A compressed cache may become more valuable when decompression is fused into kernels, transfer is overlapped with computation, placement responds to topology, or runtime policy adjusts representation according to current load.

Those are hypotheses to benchmark, not conclusions from the affinity matrix.

Choose the bottleneck before choosing the KV technique

Cognaptus would translate the survey into a constraint-first selection rule.

If GPU memory capacity is binding, compression or retention management can increase concurrency or context capacity. The evaluation still needs to include decompression cost and quality loss.

If interconnect bandwidth or transfer latency is binding, placement and migration deserve more attention than another round of representation reduction. Overlap becomes especially relevant when transfer cannot be eliminated but can be hidden behind execution.

If compute utilization or throughput is binding, scheduling, pipelining, hardware-aware execution, and device orchestration become natural candidates. A smaller cache helps only if memory pressure was preventing those resources from being used efficiently.

If user-facing SLOs are binding, the evaluation needs tail behavior rather than only mean latency. The survey identifies tail-latency reporting as comparatively sparse, which means many published optimizations provide limited evidence for precisely the regime where production operators may care most.

The same constraint-first logic applies to infrastructure purchasing. More host memory, faster interconnect, additional accelerators, or a different disaggregated serving topology will have different value depending on whether KV capacity, KV movement, or execution contention is the limiting resource.

Efficiency also changes what can go wrong

The survey treats quality degradation as a cross-cutting concern rather than a compression-only issue.

Structural methods can lose precision or remove context through compression and eviction. Spatial orchestration can fail to make required KV state available at the right time. Temporal or asynchronous execution can create inconsistent or stale handling.

The authors extend this concern to robustness, reliability, privacy, and safety, particularly when a system treats apparently low-salience context as expendable even though that context is rare but semantically critical.

This is a conceptual research agenda rather than experimentally quantified risk. Still, it changes what a production benchmark should record. For high-stakes deployments, average task accuracy after compression is not enough; evaluation should include failure under workload stress, long-context retention, migration faults, and worst-case quality behavior where relevant.

What the survey cannot tell an operator yet

The paper is a structured synthesis, not a controlled benchmark.

It introduces no new serving experiments and makes no causal claims. The authors also decline to pool speedup or memory-reduction numbers because model size, hardware, workload, implementation, and baseline choices differ too much across studies for those figures to be directly comparable.

The literature collection is broad but is not documented as a formal systematic review with database queries, screening stages, and inclusion counts. Mainstream hardware and common serving configurations dominate the evidence base, so portability to less common architectures remains uncertain.

Most importantly, the co-design affinity matrix measures what researchers have combined, not what combinations produce the highest return. A score of 0.53 is not a 53% performance improvement, nor does weak affinity prove that two techniques are incompatible.

For operators, the taxonomy is best used to generate architecture choices and benchmark requirements. It cannot replace measurement on the intended model, workload, topology, and SLO.

Serving efficiency starts with the constraint

KV cache is a memory object, but KV-cache optimization is a systems problem.

The survey’s main contribution is to make that distinction explicit. Execution timing, state placement, representation size, and retention policy affect different resources, and local improvement in one layer can disappear once decompression, transfer, scheduling, or kernel overhead enters the path.

That makes “How much KV memory did we save?” a necessary metric for some deployments, but rarely a sufficient one.

A better evaluation begins by naming the binding constraint, selecting the behavior that acts on it, and then measuring whether the entire serving path improved. The survey’s co-design results add a second question: whether a promising local technique has been integrated with the execution and movement mechanisms required to realize its benefit.

For teams already compressing KV aggressively, that may be the more productive place to look next.

Cognaptus: Automate the Present, Incubate the Future.