Give Sound a Sense of Space: Depth as a Constraint in Audio-Visual Segmentation

TL;DR for operators

A system can recognize that a sound is compatible with several visible objects and still fail to identify which object actually produced it. The missing information is sometimes spatial rather than semantic: which object is in front, where one object ends, whether two candidates overlap, and which boundary belongs to which region.

DGCM-AVS¹ tests whether estimated monocular depth can supply that missing structure. On the harder semantic audio-visual segmentation setting, removing depth from the ResNet-50 configuration reduces Jaccard/F-score from 39.6/43.5 to 37.0/40.9. With PVT-v2, the full model reaches 46.4/51.0 on the same benchmark and leads the methods reported in the paper.

The more consequential result is architectural. The model does not treat depth as another feature tensor to concatenate. One component uses smooth visual structure to preserve coherent object regions while emphasizing sharp depth changes around boundaries. Another uses depth as an intermediate representation through which audio and visual features are progressively aligned.

For perception teams, this suggests a specific design option: when audio and RGB leave several spatially plausible targets, an estimated geometry channel may constrain localization better than investing only in richer semantic fusion. That comes with higher parameter and preprocessing cost, and the paper does not establish robustness under real-world sensor noise, deployment latency, rapid source switching, or silent intervals.

Semantic agreement does not resolve spatial ambiguity

Sounding-object segmentation asks a model to identify the pixels belonging to the object producing a sound. A stronger audio encoder can improve recognition of what the sound resembles. A stronger visual encoder can improve recognition of what is visible. Neither necessarily resolves which of several compatible objects is responsible.

That distinction becomes more consequential when objects are adjacent, partially occluded, or acoustically similar. Two visible objects may both fit the audio semantically while differing in position, depth ordering, and boundary structure. In that setting, the problem is not simply weak cross-modal matching. The model lacks a geometric constraint on the candidate set.

The paper’s main intervention is to estimate monocular depth from each RGB frame using Depth Anything V2 and process it alongside RGB and audio. Because the benchmark datasets do not contain ground-truth depth annotations, this is generated geometry rather than an additional physical sensor.

The benchmark evidence is consistent with the hypothesis that geometry becomes more valuable as the scene becomes harder to disambiguate. With ResNet-50, DGCM-AVS reports 83.3/91.3 Jaccard/F-score on the single-source S4 benchmark, 61.2/74.4 on the multiple-source MS3 benchmark, and 39.6/43.5 on the more difficult semantic AVSS setting. With PVT-v2, those results rise to 85.2/92.4, 63.2/75.5, and 46.4/51.0. The paper reports the PVT-v2 AVSS result as a 10.2% relative improvement in Jaccard and 8.7% in F-score over the previous strongest compared values.

Those are benchmark comparisons, not evidence that depth will improve every deployed audio-visual system. But the pattern makes the geometric hypothesis more credible precisely where multiple plausible objects and clutter create more room for spatial confusion.

The gain is not explained by adding one more modality

A natural reading of these results is that more input information produces better predictions. The ablations make that interpretation too weak.

Removing depth drops the ResNet-50 model from 83.3/91.3 to 81.1/90.0 on S4 and from 39.6/43.5 to 37.0/40.9 on AVSS. That supports the value of depth inside this architecture. But replacing the paper’s specialized depth-processing mechanism with direct visual-depth addition also reduces performance, particularly on AVSS.

The model first separates two spatial roles. Smooth, locally aggregated visual features are used to maintain consistent object regions. Depth is treated differently: subtracting a locally smoothed depth representation from the original exposes sharper geometric changes around discontinuities, boundaries, and occlusion breaks.

The paper calls the mechanism that combines these signals the Depth-Aware Dynamic Modulator (DADM). Its purpose is not to make depth and RGB interchangeable. It preserves region-level visual coherence while allowing depth changes to sharpen distinctions between neighboring structures.

A window-size comparison functions as a sensitivity test rather than a second central claim. Among the tested settings, a low-frequency visual window of 5 and high-frequency depth window of 3 performs best, reaching the full model’s 83.3/91.3 on S4 and 39.6/43.5 on AVSS. The result supports the selected implementation, but the narrow set of tested windows does not establish a generally optimal spatial decomposition.

Depth also changes where cross-modal fusion happens

The second architectural choice addresses a different ambiguity. Directly matching an audio representation against visual features requires the model to jump from sound semantics to spatial target selection in one step.

DGCM-AVS instead inserts depth between them.

Its Depth-Guided Progressive Fusion (DGPF) first maps audio information into a depth-informed spatial representation. The resulting audio-depth representation then participates in a target-search stage that aligns audio, visual, and depth information. In operational terms, geometry helps narrow where the sound could plausibly originate before the model resolves the corresponding visual object.

The DGPF ablation supports this staged design. A variant that performs direct audio-visual target search without depth as the bridge reaches 38.7/42.4 on AVSS, compared with 39.6/43.5 for the full model. Removing the Feature Calibrator gives 39.2/43.0, while removing audio drops the result to 38.0/41.8.

These differences are not formal causal estimates: the paper reports neither repeated-seed uncertainty nor confidence intervals. They are component ablations showing that the full integration design performs better than the tested alternatives under the reported training setup.

Better segmentation comes with a system cost

The additional geometric channel is not free.

In the reported ResNet-50 efficiency comparison, DGCM-AVS contains 224.3 million parameters, versus 150.9 million for AVSegFormer and 90.7 million for AVSBench. It requires 183.4G FLOPs and reports 60.7 ms inference time, compared with 210.1G and 176.5 ms for AVSegFormer, and 166.6G and 17.7 ms for AVSBench.

So the engineering tradeoff is not simply accuracy versus latency. DGCM-AVS is faster and uses fewer FLOPs than the reported AVSegFormer configuration while carrying substantially more parameters. Against the lighter AVSBench baseline, it is both larger and slower.

Depth generation also sits upstream of segmentation. The depth-generator comparison uses a two-stage extraction-then-AVS pipeline because the strongest pretrained depth models have meaningful resource requirements. Depth Anything V2 produces the best downstream results among Depth Pro, Depth Anything, and the original Depth Anything, reaching 39.6/43.5 on AVSS versus 37.7/41.7 and 37.6/41.7 respectively.

For a product team, that makes depth quality, preprocessing capacity, parameter storage, and latency part of the architecture decision rather than implementation details to consider later.

Geometry does not solve temporal attribution

The clearest boundary appears in the paper’s failure analysis.

DGCM-AVS can preserve contours while still assigning them to the wrong sounding object. Reported failures include acoustically similar sources, a sounding object moving out of view, the sound stopping, and the active source changing quickly. The authors connect some of these errors to strong temporal dependence in continuously aligned audio-video inputs, which can leave predictions biased toward an object that was sounding previously.

This matters because spatial and temporal reliability are different problems. Estimated depth can help determine where objects are and where their boundaries lie. It does not by itself determine whether an object is producing the sound now.

The current benchmarks also under-represent some of these unusual temporal patterns, and the study provides no evaluation under real-world sensor noise or embodied perception-action conditions. There are also no significance tests, confidence intervals, or repeated-seed variance estimates around the benchmark differences.

For sensor fusion, geometry is a design variable

The business implication is narrow enough to be actionable.

For teams building multimodal perception systems, the relevant decision is whether ambiguous audio-visual localization justifies spending compute and model capacity on an explicit structural representation. DGCM-AVS provides benchmark evidence that estimated depth can be useful when semantic compatibility leaves several plausible spatial targets, especially in harder multi-source and semantic segmentation settings.

The stronger interpretation is not that every audio-visual model needs depth. It is that cross-modal fusion architecture should reflect what each modality contributes. RGB can supply object semantics, audio can identify candidate sources, and estimated depth can constrain spatial structure. The paper’s ablations suggest those roles are more productive when they are modeled explicitly rather than merged indiscriminately.

Production evaluation should therefore separate spatial ambiguity from temporal attribution. Tests for adjacent and occluded objects probe whether geometry helps. Tests for silence, source switching, similar voices or alarms, and objects leaving the frame probe a different failure surface that this architecture does not remove.

DGCM-AVS strengthens the case for treating geometry as part of multimodal reasoning rather than as preprocessing decoration. Its benchmark gains are substantial enough to justify experimentation. Its failure cases are specific enough to show what that experiment still needs to measure.

Cognaptus: Automate the Present, Incubate the Future.