Graph Minds & Gaussian Time: Why SHRIKE Rewrites Audio‑Visual Reasoning

Sound is messy. Video is messy. Put them together in a real business environment—a factory floor, a training room, a retail aisle, a vehicle cabin—and the usual fantasy of clean perception quietly dies in a corner.

A camera can see a person holding a tool. A microphone can hear a machine alarm. But the useful question is rarely “what objects exist?” or “what sound is present?” It is more awkward: which thing made the sound first? Where is the loudest source? Was the visible action actually producing the audio event, or merely happening near it?

That is the territory of audio-visual question answering, or AVQA. The paper behind SHRIKE—Multi-Modal Scene Graph with Kolmogorov–Arnold Experts for Audio-Visual Question Answering—does not solve every messy perception problem. It does something narrower and more interesting: it argues that audio-visual reasoning needs explicit relational bookkeeping before temporal reasoning can become reliable.¹

That sounds less glamorous than “just use a bigger multimodal model.” Good. Glamour is how systems become expensive demos.

SHRIKE’s core bet is simple: before asking a model to answer a question about a complex audio-video scene, first make it build a structured account of what exists, where it is, who is interacting with what, and which audio relation appears to hold. Then, when a question arrives, select the relevant relation triplets and use a sharper temporal mechanism to decide which moments matter.

The paper reports state-of-the-art results on MUSIC-AVQA and MUSIC-AVQA v2.0. The gains are real, but modest. That is exactly why the mechanism matters more than the headline number.

SHRIKE is not “an LLM watches a video and answers”

The easy misconception is that SHRIKE wins because it uses a stronger multimodal large language model. It does not, at least not in the way a casual reader might assume.

MiniCPM-o is used to generate coarse multi-modal scene graph annotations. It is not the final reasoning engine answering the benchmark questions directly. In fact, when MiniCPM-o is evaluated directly on MUSIC-AVQA, the paper reports only about 40% accuracy, far below the supervised SHRIKE model’s 78.14% on the full MUSIC-AVQA benchmark.

That distinction matters. SHRIKE is not an argument for replacing task-specific systems with a general-purpose multimodal chatbot. It is an argument for using a multimodal model as a structured annotation assistant, then training a dedicated architecture around those structures.

The workflow is closer to this:

audio + video
    ↓
coarse relation graph per segment
    ↓
question-conditioned triplet selection
    ↓
audio-video-text fusion
    ↓
Gaussian KAN temporal experts
    ↓
answer prediction

This is less magical, and much more operationally useful.

A business system that can expose intermediate relations—“person plays flute,” “piano left of violin,” “instrument louder than another instrument”—is easier to inspect than a system that merely emits an answer. Of course, easier to inspect does not mean automatically correct. SHRIKE’s own failure cases make that painfully clear. More on that later.

The first mechanism: turn perception into relation triplets

The first contribution is the Multi-Modal Scene Graph module, or $M^2SG$. Traditional scene graphs represent objects and visual relations: subject, predicate, object. For example:

person — holds — violin
piano — left of — cello

SHRIKE extends this idea into audio-visual space. For the MUSIC-AVQA setting, the authors define 24 object categories: 22 musical instruments plus person and scene. They also define six predicate categories:

This vocabulary is deliberately small. That is not a weakness by itself. In fact, it is one reason the method is interpretable. The system is not asked to invent every possible relation in the universe; it is asked to structure the relations that matter for a specific benchmark.

The scene graph is generated per temporal segment. MiniCPM-o 2.6 is prompted to extract objects and relation triplets from sampled audiovisual clips. The authors then train a Scene Graph Decoder to predict these triplets from fused audio-visual features, using Hungarian loss to handle the fact that triplets are sets rather than ordered lists.

A small but important design choice: the scene graph decoder does not use the question as input. The authors explicitly argue that scene graph prediction should remain independent of the question, because injecting question features too early could bias the graph toward what the system thinks the question wants.

That is a good instinct. In business language, SHRIKE separates observation from query. First build the event ledger; then ask the business question.

The second mechanism: select relations after the question arrives

A full scene graph can contain many triplets. Most are irrelevant to a particular question.

If the question is “Which instrument sounds first?”, spatial triplets may help only indirectly. If the question is “Where is the loudest instrument?”, volume and position relations matter more. If the question is “Is the flute always playing?”, the relevant evidence lies across time, not just inside one frame.

SHRIKE handles this through relationship triplet selection. It computes attention between the question feature and the relationship triplet features, then keeps the top-$k$ relevant triplets for each segment.

This is where the system starts to look less like generic multimodal fusion and more like a structured reasoning pipeline. The question does not merely get concatenated with audio and video features. It acts as a selector over relation candidates.

The ablation evidence supports this being more than decorative architecture. Without both proposed modules, the baseline average accuracy is 76.43%. Adding $M^2SG$ alone raises it to 77.56%, a 1.13-point improvement. Adding KAN alone raises it to 77.33%. Combining both reaches 78.14%.

That pattern is useful. The graph module is not an interpretability garnish pasted on top of the model after the fact. It contributes the larger measured gain in the main ablation.

There is a subtle point here. The visual QA score with both modules, 86.09, is slightly below the 86.13 achieved by $M^2SG$ alone. The overall result still improves because audio and audio-visual reasoning improve enough to compensate. That is not a scandal. It is a reminder that architectural changes rarely move every subtask upward in perfect harmony. Benchmark tables are not motivational posters.

The third mechanism: Gaussian KAN experts make time less blurry

The second major contribution is the Gaussian KAN-based Mixture-of-Experts module.

Previous methods such as QA-TIGER use Gaussian experts to model temporal relevance. The basic intuition is sensible: a question often depends on specific video moments. Some questions need the beginning, some the middle, some a narrow event, some a broad interval.

SHRIKE keeps the Gaussian temporal idea but replaces MLP-based experts with Kolmogorov–Arnold Network experts. KANs replace ordinary fixed activation-plus-linear-weight transformations with learnable univariate functions, often implemented with spline-like basis functions. In SHRIKE’s formulation, the KAN expert transformation can be read as a more flexible function approximator applied inside the temporal integration module.

The paper’s practical claim is not “KANs are universally better than MLPs.” The actual claim is narrower: in this AVQA temporal integration setting, KAN-based experts produce sharper, more question-aligned temporal weights than the MLP counterpart.

The qualitative figures support this interpretation. In visualized examples, SHRIKE’s blue Gaussian attention curves identify relevant segments earlier or more cleanly than QA-TIGER’s red curves. For one visual question, QA-TIGER detects one instrument but misses the second instrument in the visual stream, while SHRIKE captures both. For an audio-visual question about whether an instrument is always playing, SHRIKE identifies relevant visual and audio cues earlier, which changes the answer.

Those figures are qualitative evidence. They explain mechanism; they are not a second benchmark. The correct reading is: the visualization helps us understand why the model may improve, while the accuracy tables tell us how much it improves.

The “how much” remains modest. On MUSIC-AVQA, SHRIKE reaches 78.14% average accuracy versus 77.56% for QA-TIGER. On MUSIC-AVQA v2.0, its reported overall gains over QA-TIGER are small across biased and balanced settings: 77.33% versus 77.08%, about 76.97–76.98% versus 76.71%, 74.44% versus 74.35%, and 76.45% versus 76.08%.

That does not make the paper unimportant. It makes it more interesting for product thinking. The question is not whether SHRIKE creates a spectacular jump. It does not. The question is whether it identifies a design direction that is more robust than blindly scaling embeddings.

The evidence is a stack, not a single victory lap

The experimental section has several different kinds of evidence, and they should not be treated as equal.

The last row deserves special discipline. The transfer result on the AVE dataset is interesting, but the improvement over CACE-Net is 0.13 percentage points. That is better treated as a hint of portability, not a business case.

The real business lesson is structured perception, not benchmark worship

For Cognaptus readers, the business relevance is not “use SHRIKE tomorrow.” The paper is trained and tested on music performance videos, not factories, hospitals, classrooms, shopping malls, or warehouses.

The transferable idea is architectural: convert raw multi-modal signals into a structured event representation before asking downstream questions.

A production system for industrial monitoring would not use piano, cello, and louder. It might need relations such as:

operator — approaches — machine
machine — emits — alarm
tool — contacts — surface
worker — wears — helmet
sound_source — near — conveyor

A training-review system might need:

trainee — performs — step
supervisor — interrupts — trainee
voice_instruction — overlaps_with — action
object — placed_left_of — marker

A retail analytics system might need:

customer — picks_up — product
staff — speaks_to — customer
product — returned_to — shelf
announcement — louder_than — background_music

The vocabulary changes. The design pattern survives.

Domain-specific relation vocabulary
        ↓
Audio-video segment annotation
        ↓
Scene graph decoder trained to reproduce relation structure
        ↓
Question or policy selects relevant triplets
        ↓
Temporal reasoning identifies when the evidence matters
        ↓
Decision, answer, alert, or explanation

This matters because many business video-AI failures are not failures of object detection alone. They are failures of relation and timing.

A camera may detect a worker and a machine. The useful question is whether the worker was operating the machine when the alarm sounded. A microphone may detect a sound. The useful question is whether that sound came before, during, or after a visible action. A multimodal embedding may encode all of this somewhere in vector space. Somewhere. Very comforting. Also not enough when a compliance team asks what happened.

Structured intermediate representations can make systems more auditable, easier to debug, and more adaptable to domain-specific workflows. That is the real ROI pathway: not just higher accuracy, but cheaper diagnosis of wrong answers.

The limitation that bites: relation quality is now a bottleneck

SHRIKE’s strength is also its constraint. If reasoning depends on relation triplets, then bad or missing triplets can derail the answer.

The paper’s qualitative failure case makes this concrete. For the question “Where is the loudest instrument?”, SHRIKE preserves some relevant position and volume-related triplets but still answers incorrectly. The authors attribute the error to missing comparative loudness relations, especially for less salient instruments. In plain terms: the system may know where things are, and it may know some things are sounding, but it can still fail when the key relation is a subtle comparison.

That is not a minor edge case. Many real business questions are comparative:

• Which machine became louder after maintenance?
• Which speaker interrupted first?
• Which station was active longer?
• Which worker handled the object before the incident?
• Which alarm source dominated the scene?

Comparative relations are often long-tail, context-dependent, and harder to annotate than simple object presence. If the graph misses them, downstream reasoning inherits the blindness.

This is why scene-graph systems should not be sold as “explainable AI” without qualification. They are explainable only with respect to the relations they can reliably detect. A neat graph of wrong relations is not transparency. It is a well-formatted mistake.

The production boundary: small gains, domain vocabulary, and latency

SHRIKE is promising, but its deployment boundaries are clear.

First, the benchmark domain is musical performance. MUSIC-AVQA contains thousands of videos and tens of thousands of QA pairs, but the world is still constrained: instruments, performers, locations, sounds, and music-related questions. MUSIC-AVQA v2.0 improves evaluation with biased and balanced subsets, but it remains a music AVQA benchmark.

Second, the relation vocabulary is hand-designed and domain-specific. This is exactly what makes the method interpretable, and exactly what makes transfer non-trivial. A warehouse, hospital, classroom, or vehicle cabin would need its own object categories, predicate categories, annotation rules, and validation workflow.

Third, pseudo-annotation quality matters. The paper compares MiniCPM-o and Qwen2.5 Omni on a 2,000-video annotation setting and chooses MiniCPM-o because it performs slightly better and is much faster—about 5 seconds per video versus about 30 seconds. That is an implementation decision, not a universal law. In another domain, the best annotation model may differ, and the cost of wrong pseudo-labels may be higher.

Fourth, inference and training cost are not free. The supplementary material reports roughly two hours of training on MUSIC-AVQA using a single NVIDIA RTX 3090, with inference around 33 seconds on that GPU at batch size 32. That is reasonable for research, but production use would still need latency engineering, batching strategy, model compression, and monitoring.

Finally, the reported benchmark gains are incremental. On the original MUSIC-AVQA dataset, the improvement over QA-TIGER is 0.58 points in average accuracy. On MUSIC-AVQA v2.0, the margins are also small. The right conclusion is not “SHRIKE changes everything.” The right conclusion is “structured audio-visual relations and sharper temporal experts appear to move the frontier, and the design is worth studying.”

Small gains can still matter when they reveal a better architecture. They just should not be dressed up as a revolution wearing a lab coat.

What Cognaptus would take from SHRIKE

For business automation, SHRIKE suggests three design principles.

First, build an event ledger before building an answer engine. In messy audio-video environments, raw embeddings may be powerful but opaque. A graph of entities and relations gives the system something closer to an operational memory.

Second, keep observation and questioning separate. SHRIKE’s scene graph decoder is trained without question features, then the question selects relevant triplets later. That separation reduces the risk that the system “sees” only what the prompt hints at.

Third, treat time as a reasoning object, not just a sequence dimension. The Gaussian KAN experts are interesting because they make temporal focus question-sensitive and more localized. In business terms, this helps answer not only “what happened?” but “when did the evidence become relevant?”

Here is the compact version:

This is not a plug-and-play product architecture yet. But it is a useful blueprint for teams building audio-video intelligence systems that need more than object tags and transcript snippets.

Conclusion: the graph is the memory, the Gaussian is the clock

SHRIKE’s most important contribution is not that it beats prior methods by a fraction of a percentage point. The useful lesson is architectural.

Audio-visual reasoning needs memory: explicit relations among objects, actions, positions, and sounds. It also needs a clock: a way to decide which moments matter for the question being asked. SHRIKE gives the model both—a multi-modal scene graph for structured memory, and Gaussian KAN experts for question-guided temporal focus.

The paper is not a final answer for real-world surveillance, robotics, training review, retail analytics, or industrial monitoring. Its evidence is still centered on musical AVQA benchmarks. Its gains are incremental. Its failure cases show that comparative audio relations remain hard.

But the direction is correct. In complex audio-video environments, the winning systems will not merely “look and listen.” They will keep relational accounts of what they saw and heard, then reason over those accounts in time.

That is less flashy than a giant model confidently guessing from a blob of embeddings.

Which is precisely why it might work.

Cognaptus: Automate the Present, Incubate the Future.