Better Boxes, Unproven Specialists

TL;DR for operators

A visual search system may detect every object in an image yet still return the wrong one when a user describes a particular target. LaMER¹ addresses that selection problem by combining four pretrained visual encoders through language-conditioned routing, without using annotated target bounding boxes during its own training.

The reported system achieves 77.67% on RefCOCO, 62.81% on RefCOCO+, and 69.49% on RefCOCOg validation, measured as the percentage of predicted boxes reaching at least 0.5 intersection-over-union with the annotated target. More revealingly, adding cross-attention routing to an existing four-expert configuration improves accuracy by 2.5 and 4.6 percentage points on RefCOCO and RefCOCO+, respectively.

The results support adaptive feature fusion as an accuracy improvement. They do not establish that different language requests consistently activate different visual specialists: EfficientSAM receives the highest routing weight in all eight illustrated queries. Nor does this mixture-of-experts design establish computational savings, because all four experts contribute and no inference-cost comparison is reported.

For product teams, LaMER provides a reason to benchmark adaptive fusion—not yet a complete justification for deploying four additional encoders.

Finding the right object requires more than detecting objects

Consider a warehouse image-search interface receiving the request, “Find the blue package with the white label.” A detection model may identify several packages correctly. The remaining problem is determining which detection matches the description.

Training that matching behavior ordinarily becomes easier when each expression is paired with an annotated target box. But collecting those annotations for many objects and language variations creates an additional data requirement.

Weakly supervised referring expression comprehension addresses this problem by learning the association between descriptions and image regions without using ground-truth target boxes as training supervision. It does not eliminate reliance on pretrained visual models, which may themselves have benefited from supervised learning.

LaMER approaches the task by retaining a detector, incorporating several visual representations, and learning how to combine their information according to the image and the expression.

The critical distinction is between possessing multiple encoders and using their representations effectively.

How language-conditioned routing changes the visual representation

LaMER uses YOLO11x to produce candidate object regions and multi-scale visual features. A bidirectional GRU and attention layers encode the referring expression into a contextual language representation.

Four frozen visual encoders provide additional information:

• SigLIP supplies visual-language semantic representations.
• DINOv2 contributes self-supervised visual structure.
• EfficientSAM supplies segmentation-related spatial information.
• ConvNeXt contributes appearance and texture representations.

Their features are projected into compatible dimensions and aligned spatially before fusion.

Rather than assigning fixed contributions to the four encoders, LaMER constructs a routing query from the pooled YOLO11x visual representation and the encoded language expression. Cross-attention then produces four normalized expert weights.

The weights combine the spatial expert features. A second mixing rule controls how strongly that combined representation influences the YOLO11x backbone: when one expert receives a more concentrated weight, the expert mixture receives greater influence relative to the backbone. A learnable residual gate provides further adjustment.

This architecture allows the representation used for localization to depend on both the image and the requested object.

After fusion, the system selects 17 promising cells from a 13×13 grid using detection objectness. A cell-text matching head scores these candidates against the expression and selects the corresponding detection box.

Training combines contrastive anchor-text matching, additional negative anchors from other batch samples, language-feature reconstruction, cross-modal alignment, and expert load-balancing regularization. Although the mathematical objective includes an optional supervised bounding-box loss, its coefficient is zero in every reported experiment.

The architecture therefore trains its language-guided selection mechanism without direct target-box regression supervision.

Routing adds accuracy beyond simply including four encoders

The clearest evidence for routing comes from LaMER’s cumulative component ablation, rather than its comparison with entirely different systems.

Source: Section 4.5.1, Table 3. These are cumulative architecture ablations, not independently randomized component combinations.

Adding the router produces gains of 2.5 percentage points on RefCOCO and 4.6 points on RefCOCO+ while retaining the four-expert configuration. This is evidence that the fusion mechanism itself contributes to localization accuracy.

The later changes are less uniform. Adding multi-scale fusion raises RefCOCO validation accuracy but temporarily lowers RefCOCO+ performance from 60.8% to 60.2%. The subsequent bottleneck projection raises it to 62.3%, while selectively fine-tuning the YOLO11x attention block supplies a smaller final gain.

These results support the full configuration but do not establish that every component improves every setting independently.

The main benchmark comparison provides a second perspective. Against SVCR’s stronger YOLOv5 configuration, LaMER reports validation results of 77.7% versus 74.5% on RefCOCO, 62.8% versus 56.4% on RefCOCO+, and 69.5% versus 60.4% on RefCOCOg. These are accuracy improvements of approximately 3.2, 6.4, and 9.1 percentage points.

RefCOCO+ is particularly relevant because its descriptions exclude spatial-location words, placing greater emphasis on appearance-based identification. RefCOCOg uses longer descriptions.

Those comparisons support improved benchmark performance, but they are not fully controlled architectural comparisons: the systems differ in backbones and training arrangements, and some baseline values come from earlier publications or repositories.

More experts help, but the router does not prove specialization

The appendix tests whether additional visual representations contribute beyond a single encoder.

In the expert-combination ablation, YOLO11x alone reaches 67.2% RefCOCO validation accuracy. Combining it with DINOv2 raises accuracy to 74.2%. Subsequent combinations incorporating SigLIP, EfficientSAM, and ConvNeXt improve the reported results further.

A separate top-N routing ablation shows an especially large difference between retaining one expert and retaining more. RefCOCO validation accuracy increases from 38.1% with Top-1 to 75.5% with Top-2, 77.1% with Top-3, and 77.7% with Top-4. RefCOCO+ validation follows the same direction, rising from 29.2% to 62.8%.

These are ablation results within LaMER’s tested configurations. They establish neither that arbitrary additional encoders will help nor that four experts are economically preferable to two.

A more consequential uncertainty concerns what the learned router actually does.

LaMER includes a load-balancing penalty that encourages average routing weights across a training batch toward 25% per expert. This discourages excessive concentration but does not require equal weights on each query.

Appendix C.3 reports routing weights for eight illustrated examples. Their approximate averages are:

• EfficientSAM: 49%
• SigLIP: 28%
• DINOv2: 18%
• ConvNeXt: 5%

EfficientSAM receives the highest weight in every illustrated example.

The examples show that expert utilization remains unequal despite regularization. They do not establish that EfficientSAM dominates across the entire dataset, since eight selected examples are not a representative distributional analysis.

More importantly, improved routing accuracy does not prove that the mechanism consistently assigns different expressions to interpretable visual specialists. It may be benefiting from a generally stronger representation, conditional adjustments around a dominant expert, or interactions that these measurements do not isolate.

That distinction matters when an engineering team considers whether all four encoders are necessary for its particular requests.

A valid candidate can still become the wrong answer

The paper’s qualitative examples reveal a failure that aggregate accuracy can obscure.

Seven of eight illustrated predictions correctly localize their targets. In the unsuccessful example, the system selects the wrong person even though the correct target is available among the candidate boxes. The reported intersection-over-union for that prediction is zero.

This separates two failure mechanisms: generating a suitable candidate and choosing the correct candidate from those already available.

For a warehouse or retail search application, improving detection coverage would not necessarily fix the second problem. A team evaluating language-guided item identification needs to measure incorrect selection conditional on the correct object being present among proposals.

The benchmarks measure localization overlap, not downstream business outcomes or physical task success. They also provide point estimates without reported confidence intervals or repeated-seed variability.

The production decision remains an accuracy-versus-cost comparison

Cognaptus interprets LaMER as evidence for testing query-conditioned feature fusion when an application must locate described objects and target-box annotation is limited.

The appropriate comparison for a visual-search product is not merely LaMER against older published benchmarks. Under the product’s actual camera conditions, item categories, language distributions, and hardware budget, an engineering team would need to compare a simpler detector-based system, static multi-encoder fusion, and adaptive routing.

The measurements should include localization accuracy, errors when the correct candidate already exists, sensitivity to unfamiliar descriptions, and performance as the image environment changes. Latency, GPU memory, throughput, and per-request inference cost are equally necessary to determine whether the measured accuracy improvement warrants the additional encoders.

LaMER reports training on two RTX 5090 GPUs, but does not quantify inference latency, memory use, energy consumption, or cost relative to simpler alternatives. Its routing is soft: all four experts contribute rather than only a computationally sparse subset. Consequently, the mixture-of-experts terminology should not be read as evidence of reduced computation.

The paper also provides no production evaluation in robotics, navigation, medical imaging, or warehouse systems. Its results are confined to the reported referring-expression benchmarks.

These limitations do not invalidate the accuracy findings. They define the additional evidence needed before making a system-selection or investment decision.

Better localization is established; the best architecture is not

LaMER demonstrates that language-conditioned fusion of heterogeneous visual representations can improve weakly supervised object localization. The router’s contribution is visible even after four experts have already been included, making the method of combining representations a substantive part of the result.

The unresolved questions concern how consistently the experts specialize, whether their added computational burden is justified, and how reliably the system selects the correct object outside benchmark conditions.

For operators, the strongest conclusion is specific: adaptive multi-expert routing deserves comparison against simpler grounding architectures when target-box annotations are scarce, but its deployment value must be established through application-specific accuracy, failure, and cost measurements.

Cognaptus: Automate the Present, Incubate the Future.