Choose the Bottleneck Before the KV Cache Strategy
TL;DR for operators Mamo, Kogiou, Yi, and Yu compare three representative approaches to managing the memory accumulated during LLM generation: keep the full cache on the GPU, permanently discard selected cached tokens, or retain the larger cache in CPU memory and fetch selected entries during decoding.1 Their benchmark finds no dominant strategy. ...