Not Every Layer Deserves the Same Cache

TL;DR for operators

A compressed KV cache creates two separate decisions: which tokens to retain inside each Transformer layer, and how much of the total cache budget each layer should receive. The second decision is easy to hide behind uniform allocations or simple depth schedules, but the evidence here suggests that those rules can spend scarce GPU memory in the wrong places.

RippleKV¹ profiles each model offline by perturbing one layer’s Value cache at a time and measuring how much the final predictive distribution changes. That response correlates much more strongly with isolated compression damage than layer position does: mean absolute Spearman correlation is 0.799 versus 0.359. RippleKV then converts the sensitivity profile into unequal per-layer budgets while keeping the global cache budget fixed.

Across the five compressed model-budget settings reported in detail, RippleKV has the highest aggregate LongBench score. The practical implication is narrower than “better KV compression”: for teams already compressing KV state, model-specific allocation across layers may recover quality without changing model weights or adding online profiling. The unresolved operational cost is the offline profiling itself, because the paper does not report the calibration-set size, provenance, or measured profiling expense.

A fixed cache budget does not imply equal layer budgets

Long-context inference eventually turns memory into an allocation problem. Once an operator decides that only 10%, 20%, or 30% of the original KV cache can be retained, that capacity still has to be distributed across the model’s layers.

Uniform allocation assumes that the layers tolerate compression similarly. A depth-based schedule makes a different assumption: architectural position provides enough information to decide which layers deserve more cache. The paper tests that premise by compressing one layer at a time while leaving the others untouched and measuring how much the model’s final output distribution changes.

For Llama-3.1-8B-Instruct’s 32 layers, isolated compression damage ranges from 0.0030 to 0.0524 and reaches its maximum at an intermediate layer. Layer index has only a 0.359 mean absolute Spearman correlation with that damage, with correlations ranging from 0.218 to 0.480.

That result changes the optimization target. KV compression is not only about identifying valuable tokens. It is also about deciding where the compressor should be allowed to retain more of them.

RippleKV measures damage propagation instead of inferring importance from depth

RippleKV keeps the base compressor’s within-layer token-selection mechanism; in the reported implementation, that mechanism is SnapKV. Its contribution sits one level above token scoring.

During offline profiling, the method perturbs the prefix Value cache of one compressible layer while leaving its Key cache and every other layer unchanged. The injected Gaussian noise is scaled by each Value vector’s norm, reducing the extent to which raw activation magnitude determines the intervention.

The system then asks how much the model’s final predictive distribution changes. That change is measured with KL divergence: here, simply a numerical measure of how far the perturbed output distribution moves from the full-cache reference. Averaging this response across output positions and calibration examples produces a sensitivity score for each layer.

This signal is not actual compression damage. It is a proxy intended to predict which layers are more vulnerable before selecting a particular per-layer budget. The diagnostic evidence supports that role: its mean absolute Spearman correlation with isolated compression damage is 0.799, with a reported range of 0.723-0.857, more than twice the mean correlation obtained from layer index.

The scores are normalized and converted through an exponential weighting rule into larger or smaller cache allocations. The weights are clipped and renormalized so their mean remains one. In operational terms, RippleKV moves memory between layers; it does not increase the total budget.

Matched-budget results support the allocation mechanism

The main benchmark evidence compares RippleKV with StreamingLLM, H2O, SnapKV, and PyramidKV under the same total cache budget.

For Llama-3.1-8B-Instruct, RippleKV reaches average LongBench scores of 35.07, 40.35, and 42.97 at 10%, 20%, and 30% cache retention. The strongest competing baseline scores 34.24, 39.93, and 42.26 respectively, giving RippleKV leads of 0.83, 0.42, and 0.71 points.

At 10% retention, RippleKV also leads the reported aggregate results for Qwen2.5-7B-Instruct at 34.42 and Mistral-7B-Instruct-v0.3 at 33.19. The margins over the strongest reported baseline are 0.28 and 0.65 points.

These are not dramatic absolute gains, and RippleKV does not win every individual dataset. Their significance is architectural: the total memory allowance and within-layer scoring logic are held fixed while the layer allocation changes. The benchmark improvement is therefore evidence that where cache capacity is placed contributes independently to compressed-model quality.

The ablation separates unequal budgets from the sensitivity signal

The ablation study serves a different purpose from the main benchmark comparison: it tests which parts of RippleKV contribute to its advantage.

Removing layer-wise budget allocation and returning to uniform budgets lowers Code performance by 3.42 points, Synthetic tasks by 2.25, and Multi-Document QA by 1.37. Few-shot Learning is the exception, where uniform allocation is 0.08 points higher.

A second ablation keeps the allocation procedure but replaces final-output sensitivity with a layer-local response. Code falls by 3.18 points, Synthetic by 2.00, and Multi-Document QA by 1.54.

Together, these tests support two distinct claims. Unequal layer budgets matter, and the downstream response of the whole model provides a more effective allocation signal than observing a layer locally.

The allocation-ratio experiment is better read as a robustness check. Changing $R$ from 1.25 to 1.50 to 1.75 produces aggregate scores of 34.67, 35.07, and 34.89. A maximum spread of 0.41 points suggests that the reported result is not dependent on a single sharply tuned value within this tested range.

Offline profiling keeps the online path simple

A sensitivity method would be less attractive operationally if every request required extra perturbation runs. RippleKV avoids that by profiling the model offline and reusing the resulting sensitivity pattern when applying different global cache budgets.

The measured online efficiency is consequently close to the other compressed methods. At 128K context, RippleKV reports 43.02 seconds of latency and 42.99 tokens per second, compared with 43.87 seconds and 40.92 tokens per second for SnapKV and 43.87 seconds and 42.51 tokens per second for PyramidKV.

Peak memory falls from 45.53 to 31.13 GiB relative to Full Cache at 128K context, and from 76.09 to 47.29 GiB at 256K. Those reductions should not be attributed specifically to RippleKV’s allocation rule: they primarily reflect operating with a compressed cache. RippleKV’s claim is that the fixed compressed capacity can be distributed more effectively.

The operational decision is when to profile a model

For an inference platform team, the paper suggests a concrete change in deployment policy. When GPU memory is capped and KV compression is already part of the serving stack, per-layer budgets should be treated as model-specific configuration rather than inherited from a uniform rule or a generic depth schedule.

The paper directly shows that this policy improves aggregate LongBench performance in its tested configurations. Cognaptus’ operational inference is that the profile could become part of model-release preparation: a new or materially altered checkpoint would receive a new sensitivity profile, after which serving systems could select different global cache budgets without repeating the profiling for every request.

What remains unresolved is whether the economics justify that step. The paper calls the calibration set small but does not report its size or provenance, and it provides no measured offline profiling cost. Teams cannot yet compare profiling expense against the quality or density benefit from the published evidence alone.

Generalization is also bounded. The experiments cover three instruction-tuned models around 7B-8B parameters and LongBench workloads. Direct implementation evidence retains SnapKV’s token-selection rule, so compatibility with other compressors is a design claim rather than a broadly tested result. The perturbation procedure also modifies Values while holding Keys fixed, leaving sensitivity to Key-mediated attention changes outside the measured signal.

Cache compression has a placement problem

RippleKV’s more durable contribution is not a new rule for discarding individual tokens. It identifies a resource-allocation layer that sits above that decision.

When compression damage differs substantially across Transformer layers, equal capacity is no longer a neutral default. The paper provides evidence that final-output perturbation response can identify those differences better than layer depth and that using the resulting profile improves aggregate benchmark quality under the same total memory constraint.

For operators, that makes KV-cache configuration more model-specific. The remaining question for deployment is economic rather than conceptual: how much offline profiling is required to obtain a stable profile, and how often must it be repeated as models, compressors, and workloads change?

Cognaptus: Automate the Present, Incubate the Future.