Prune the History, Pack the Survivors

TL;DR for operators

Persistent audiovisual assistants cannot keep extending their working history indefinitely on consumer hardware. As earlier inputs accumulate, the model stores reusable representations so later updates do not have to recompute the entire history; those stored representations consume memory and also increase the attention work required on every update. In one reported Apple M2 Pro profile, MiniCPM-o-4.5 approaches 80K cached tokens and about 23 GiB after 6.25 minutes, while per-chunk prefill latency rises from 28 to 69 seconds and attention time rises from roughly 2 to 39 seconds.

OmniTide addresses both what history survives and how those survivors are physically organized. OmniPick keeps complete recent audiovisual units and selected structurally significant spans from older history without calculating online attention-importance scores. OmniPage then places and selectively migrates the surviving KV-cache entries so that logical deletion also reduces the physical span and attention work, rather than leaving scattered survivors across partially occupied regions.

The reported results are substantial but configuration-specific. Across 355 StreamingBench sessions on CUDA, end-to-end latency falls from 40.33 to 17.61 seconds for Qwen2.5-Omni-3B and from 51.71 to 21.52 seconds for Qwen2.5-Omni-7B, equivalent to 2.29x and 2.403x speedups. Under identical logical retention, OmniPage reduces physical KV span by 26.7% for both Qwen models and by 10.0% for MiniCPM-o-4.5.

For deployment teams, the larger lesson is methodological: evaluate task quality, bounded retention, physical cache layout, and end-to-end responsiveness together. Fewer retained tokens do not necessarily produce proportionally lower execution cost, and larger retention budgets do not reliably produce better task quality.

Streaming history becomes a compute problem as well as a memory problem

Consider an always-on assistant processing video and audio continuously. Every new segment adds information that may matter later, yet the device has a finite memory budget and must continue responding promptly. Keeping everything eventually becomes impractical; deleting history therefore appears unavoidable.

The difficulty is that accumulated history affects more than capacity. In one reported Apple M2 Pro profile, MiniCPM-o-4.5 approaches 80,000 cached tokens and roughly 23 GiB after 6.25 minutes of streaming. Per-chunk prefill latency rises from 28 to 69 seconds. Attention processing alone increases from about 2 to 39 seconds, while MLP time remains near 4.5 seconds.

The relevant stored state is the KV cache: reusable representations of earlier inputs that prevent the model from recomputing its complete history on every update. Those representations save repeated computation, but they consume memory and remain involved in subsequent attention. Sustained streaming therefore creates a joint memory-and-execution problem.

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming¹ treats the two as one design problem rather than optimizing token deletion and cache management independently.

Multimodal history is not a sequence of interchangeable tokens

A recent-token window works best when age is a reasonable proxy for usefulness. Streaming audiovisual input complicates that assumption because synchronized image, audio, and associated text form meaningful temporal groups.

OmniTide calls such a group a multimodal streaming unit. The distinction matters because deleting arbitrary tokens can break a unit while preserving fragments that are less useful on their own.

The authors’ attention analysis, covering 10 LongVALE videos for each of three analyzed models, identifies three recurring patterns: strong attention toward recent context, unit-local structural sinks, and different attention behavior across modalities. This analysis is best read as design evidence rather than a universal law about every attention head or model; the sample is deliberately limited and qualitative.

OmniPick converts those observations into a deterministic retention policy. It protects complete recent units while selectively retaining older boundary, sink, text, audio, and image spans using modality-specific budgets. Importantly, runtime selection does not require collecting online attention scores.

The quality comparison supports the combined policy. OmniPick beats unit-level FIFO and token-level sliding across all nine evaluated model-benchmark combinations. On StreamingBench, its gains over FIFO are 4.7 percentage points for Qwen2.5-Omni-3B, 2.6 points for Qwen2.5-Omni-7B, and 6.0 points for MiniCPM-o-4.5, while remaining descriptively close to full-context quality across the broader evaluation.

That does not establish that pruning inherently improves accuracy. Some bounded-history scores happen to exceed their full-context counterparts, but the experiments are not designed to identify pruning as the cause.

There is also no monotonic rule that retaining more history is better. Watermark sensitivity varies by model, and larger retention budgets do not consistently improve accuracy. A production configuration therefore needs model-specific tuning rather than a single universal context budget.

Logical sparsity still has a physical shape

Removing tokens creates another problem that is easy to miss.

Suppose a retention policy deletes most of an old region but preserves a few scattered tokens. Logically, the history is sparse. Physically, those survivors may still occupy pages or attention-processing tiles that remain active. The system can therefore retain far fewer useful KV cells without receiving a proportional reduction in memory exposure or kernel work.

OmniPage addresses this physical KV fragmentation. It groups entries according to retention state, aligns storage with attention tiles, and selectively migrates survivors when moving them reduces active pages, subblocks, or physical span. It avoids continuously repacking the complete cache.

The paper isolates this contribution with an especially informative ablation: logical retention is held constant while only physical organization changes. OmniPage reduces physical KV span by 26.7% for both evaluated Qwen models and by 10.0% for MiniCPM-o-4.5. Relative to the native layout, cumulative prefill attention GPU time falls 26.1% and decode attention time falls 22.5%.

An eager-compaction configuration provides an oracle-like comparison for what more aggressive repacking could achieve. Selective migration captures 88.5% of its prefill latency reduction and 82.1% of its decode reduction. The purpose of this test is not to establish a second retention method; it shows how much of full compaction’s benefit can be recovered without continuously copying the entire surviving cache.

This result is central to the paper’s systems argument: logical token sparsity and hardware-realized sparsity are different quantities.

The integrated system produces larger gains than retention ratios imply

At a 32K no-slide checkpoint, CUDA update latency for Qwen2.5-Omni-7B falls from roughly 535 ms to 234 ms, while attention time drops from about 133 ms to 10 ms. On Apple Metal, the same model’s update latency falls from approximately 11.9 seconds to 4.3 seconds.

The end-to-end session measurements are more useful for deployment interpretation because they include work outside the attention kernel. Across 355 StreamingBench CUDA sessions with a median context around 70K tokens:

These measurements show why kernel-level acceleration should not be translated directly into application-level speedup. Other computation remains in the workflow even when attention becomes much cheaper.

They also show why the Qwen results should not become a blanket expectation. MiniCPM exhibits smaller timing gains, and the paper does not establish that the same magnitude will carry to other architectures.

Edge deployment should test retention and layout as one configuration

The paper directly shows that, for the evaluated systems, multimodal structure can guide bounded-history retention and that retention-aware physical organization can turn more of that sparsity into measurable execution savings.

Cognaptus infers a broader deployment principle from those results: when assessing a persistent on-device assistant, do not certify the pruning policy and memory manager separately if they interact at runtime. The meaningful unit of evaluation is the combined configuration.

That changes several engineering decisions:

The training-free property is operationally relevant: OmniTide modifies inference behavior rather than requiring a newly trained sparse-attention model. That lowers one adoption barrier, but it does not eliminate systems integration or tuning work.

The evidence is strong inside a still-narrow deployment envelope

The evaluation spans three primary omni-modal models, three streaming benchmarks, and two consumer-device GPU backends: an RTX 4090 using CUDA and an Apple M2 Pro using Metal. That provides substantial comparative evidence for the tested configurations, not a universal result for edge multimodal inference.

Several boundaries matter. The attention-pattern analysis uses a limited LongVALE sample. The task-quality experiments evaluate OmniPick as a combined policy rather than isolating the causal contribution of every retained span type. H2O is timed without FlashAttention because the required attention scores are unavailable from the relevant kernels, making that runtime comparison asymmetric. vAttention is evaluated only on CUDA because of its dependence on CUDA virtual-memory facilities.

Battery consumption and thermal behavior are also not measured. Lower attention work could matter for those constraints, but the paper does not establish those effects.

Efficient streaming depends on what survives and where it lands

OmniTide’s main contribution is not another claim that LLM history can be pruned. It shows why sustained multimodal streaming benefits from connecting two decisions that are often evaluated separately: which pieces of history remain logically available to the model, and how those survivors are physically presented to the hardware.

OmniPick provides evidence that multimodal structure can guide retention without online importance scoring. OmniPage shows that the resulting sparsity must then be organized physically before much of its potential execution benefit appears.

For persistent edge assistants, that is the useful standard to carry forward: bounded context should be evaluated as a complete retention-and-layout system, using task quality and end-to-end latency rather than retained-token counts alone.

Cognaptus: Automate the Present, Incubate the Future.