Compress the Videos, Not the Construction Loop

TL;DR for operators

A much smaller video training set is useful only if creating it does not become another expensive optimization job. ProtoBlend targets that second cost: after preparing and running a teacher model, it constructs the compact set directly instead of repeatedly optimizing stored synthetic videos. On MiniUCF, its reported post-teacher construction time stays below one hour at both one and five videos per class and changes only slightly as the storage budget grows because cached teacher outputs are reused.¹ :contentReference[oaicite:0]{index=0}

The evidence separates three jobs. First, choose useful temporal evidence: teacher-guided clip selection provides the largest isolated gain, adding 8.2 percentage points at one video per class and 8.9 at five on MiniUCF. Second, use additional slots to cover different modes within a class: clustering adds another 4.2 points at five videos per class but little at one. Third, combine examples selectively: in-cluster prototype–anchor pairing beats random same-class pairing by 7.4 points at one video per class and 4.5 at five, while blending works best when the supervisory target is mixed in the same proportions as the source clips.

For teams that retrain video models repeatedly, Cognaptus interprets the practical opportunity as reusable compact training surrogates whose construction cost grows only modestly across compression tiers. The boundary matters: gains scale less well on motion-centric Something-Something V2 and heterogeneous Kinetics-400, where frame-aligned pixel interpolation can misalign motion phases and create ghosting.

The compression pipeline can cost more than the compressed set

Suppose a team wants a tiny video training set for repeated architecture tests, hyperparameter sweeps, or ablations. Reducing the number of stored clips helps each downstream training run, but conventional dataset distillation can introduce a separate cost: repeatedly optimizing synthetic video tensors to make those few examples informative.

ProtoBlend targets that construction cost. Its reported post-teacher construction time on MiniUCF remains below one hour at both one and five videos per class and changes only slightly as the target budget increases. The reason is structural: teacher scores, embeddings, and predictions are computed and cached, while a larger distilled set mainly requires new clustering, pairing, and blending rather than another round of synthetic-video optimization.

That claim has a precise boundary. ProtoBlend does not remove training or gradient computation from the workflow. A dataset-specific VideoMAE teacher is first fine-tuned, and student networks are later trained on the distilled set. What disappears is iterative gradient-based optimization of the stored distilled videos themselves.

Selection and allocation solve different compression problems

ProtoBlend first samples five temporal candidates from every source video. A frozen teacher scores them, and the candidate with the lowest target-class cross-entropy is retained. In plain language, each source video contributes the segment the teacher finds easiest to recognize correctly.

The MiniUCF ablation indicates that this is the pipeline’s strongest isolated component. Starting from random selection, teacher-guided temporal selection raises accuracy from 10.3% to 18.5% at one video per class and from 20.7% to 29.6% at five. These are ablation results, so their purpose is component attribution rather than a new benchmark claim: weak temporal segments can damage a compact video set before any sophisticated condensation step begins.

Once more than one distilled slot is available, the problem changes. Selecting only the highest-confidence clips could spend several slots on similar examples. ProtoBlend therefore clusters teacher embeddings separately within each class, with the number of clusters equal to the videos-per-class budget. Each cluster receives a slot; confidence then decides which clip becomes its prototype.

That distinction explains the budget-dependent ablation. With one video per class, there can be only one cluster, so cluster-guided allocation contributes little. At five videos per class, adding it raises MiniUCF accuracy from 29.6% to 33.8%, a 4.2-point gain. Extra storage creates value only if additional slots cover different intra-class modes rather than repeat the same one.

The main results are consistent with that mechanism. ProtoBlend reaches 20.9%, 34.8%, and 41.0% on MiniUCF at one, five, and ten videos per class. It is second at the smallest budget but first at five and ten. On HMDB51 it ranks first at all three reported budgets, reaching 9.0%, 12.1%, and 14.2%.

Blending works only when the sources and supervision remain compatible

Each cluster contributes two clips: the lowest-loss clip as prototype and the second-lowest-loss clip as anchor. ProtoBlend then constructs one stored video by frame-aligned pixel interpolation:

The important detail is not simply that two clips are mixed. The pairing control shows why locality matters. On MiniUCF, random same-class pairing reaches 13.5% at one video per class, versus 20.9% for in-cluster pairing. At five videos per class, the comparison is 30.3% versus 34.8%. This experiment is a controlled pairing test: its role is to show that same class membership alone does not guarantee a useful interpolation partner.

Supervision is constructed with the same constraint. Rather than asking the teacher to classify the potentially unusual blended video, ProtoBlend mixes the cached teacher posteriors of the original prototype and anchor using the same coefficient:

The component-interaction experiment shows that blending and soft supervision are complementary rather than independently additive. Adding the mixture-source target to hard-label blending improves MiniUCF accuracy from 19.3% to 20.9% at one video per class and from 32.3% to 34.8% at five. The useful unit is therefore not “more information through averaging,” but a compatible visual mixture paired with supervision that changes consistently with that mixture.

Reusable surrogates are the business opportunity

The paper directly establishes benchmark accuracy and post-teacher construction efficiency. The business interpretation goes one step further.

For a team repeatedly retraining video models, a distilled set is potentially reusable infrastructure: construct it once, then use it for many student-training experiments without repeatedly accessing the full source dataset or teacher. Because cached teacher outputs can also support different videos-per-class budgets, the same preparation step could support several storage or compute tiers with relatively modest additional construction work.

That interpretation is most defensible for workflows where selected clips can capture much of the useful class information and where intra-class diversity can be represented by teacher-feature clusters. The paper does not measure end-to-end organizational ROI, data-governance savings, or privacy benefits, so those remain deployment hypotheses rather than demonstrated outcomes.

Motion-heavy video exposes the current boundary

ProtoBlend is not uniformly best as the budget grows. On Something-Something V2, it leads at one video per class with 4.2% top-5 accuracy but reaches only 4.3% at five, below K-center’s 4.5%. On Kinetics-400, it narrowly leads at one video per class with 7.2%, then reaches 8.0% at five, below DM’s 9.1% and slightly below PRISM’s 8.1%.

The paper connects this weaker scaling to the construction mechanism itself. Frame-aligned interpolation does not explicitly align motion phases, viewpoints, or spatial positions. When paired clips differ along those dimensions, blending can create ghosting rather than a coherent composite training example.

Cross-architecture results are encouraging but narrower than a general transfer claim: at MiniUCF with one video per class, ProtoBlend ranks first for ConvNet3D, CNN+GRU, and CNN+LSTM. The experiment does not establish architecture independence across the other datasets or larger budgets.

ProtoBlend’s broader contribution is therefore a change in where video distillation spends computation. Its strongest evidence suggests that good compact datasets need three separate decisions: which temporal evidence is worth keeping, how scarce slots cover variation, and which examples can be combined without making the input and supervision disagree. When those conditions hold, directly constructing the distilled set can replace a substantial synthetic-video optimization loop. When fine-grained motion is the information being compressed, the blending mechanism still has work to do.

Cognaptus: Automate the Present, Incubate the Future.