Two Cameras, One Vehicle, Misaligned Edges

TL;DR for operators

Combining visible and thermal images does not automatically produce a better observation. When corresponding objects appear at different positions, fusion can introduce duplicated edges and structural blur that interfere with detection.

SAGE addresses this problem by retaining source-specific information while coordinating appearance equalization, geometric alignment, and image fusion. On the DroneVehicle benchmark, its fused images support vehicle-detection [email protected] of 16.04 versus 12.00 for the strongest competing method, with improvements in both precision and recall.

The architectural lesson is to correct global and local misalignment differently, then incorporate alignment reliability into fusion. The procurement lesson is more conditional: SAGE achieves strong structural accuracy but does not lead every metric, and its 91.45 GFLOPs require hardware-specific validation.

When two cameras disagree about a vehicle’s position

An aerial monitoring system can observe the same vehicle through a visible camera and a thermal sensor. The visible image captures appearance and edges; the thermal image contributes information derived from emitted radiation. Both observations are valuable, but their vehicle outlines may occupy slightly different positions because of viewpoint differences, parallax, or platform movement.

Combining these images without correcting their geometry can produce ghosting: two displaced representations of a structure appear in one fused image. A sharper-looking result may still contain spatial inconsistencies that make vehicle recognition harder.

This distinction becomes measurable in SAGE: Source-Anchored Guidance via Frequency Equalization for Hierarchical RGB-T Alignment and Fusion, by Timing Li and colleagues.¹

Using the DroneVehicle dataset, the researchers train and test YOLO11m on outputs from nine fusion methods. SAGE produces the strongest reported detection performance:

• [email protected]: 16.04, compared with FusionRegister’s 12.00, a difference of 4.04 points on the reported scale.
• Recall: 33.63, compared with the next-highest 27.51 from MURF.
• Precision: 19.31, compared with FusionRegister’s 16.23.

These results establish a downstream benefit under the tested detection protocol. They also motivate a more specific engineering question: which information must survive the image-processing pipeline for the final observation to remain geometrically coherent and useful to a detector?

Source information must survive appearance equalization

Visible and thermal images differ not only in position but also in how they represent the scene. A structure that is visually prominent may have a relatively weak thermal signature, while a thermal contrast may not correspond to a strong visible edge.

SAGE’s principal architectural contribution is to coordinate these differences rather than process appearance normalization, registration, and fusion as isolated operations.

The system first applies a Haar wavelet decomposition, separating each image into a low-frequency approximation and three directional high-frequency detail components. Low frequencies represent broader image characteristics; high frequencies capture localized structural detail.

Its source-anchored frequency equalization module, SAFE, then uses an invertible joint encoder and source-specific gain predictors to adjust the low-frequency representations. These adjustments are bounded by a modulation parameter of 0.2 in the reported experiments.

Crucially, SAFE leaves the original high-frequency coefficients unchanged during equalization. The system can reduce low-frequency appearance discrepancies without directly overwriting the source-specific detail coefficients at this stage.

The joint representation also produces structural guidance and gain guidance, denoted $Q_s$ and $Q_g$. Rather than discard these intermediate signals, SAGE carries them into subsequent alignment and fusion operations.

This creates a dependency across stages: information acquired while making the sensors easier to compare also informs how they are spatially matched and how their observations are combined.

The paper’s architecture establishes this mechanism. Whether the information-sharing design actually contributes to performance requires the separate ablation and guidance-routing experiments discussed below.

Global geometry and local displacement require different corrections

A single registration mechanism must otherwise handle two distinct problems: correcting the overall spatial relationship between sensors and resolving residual local mismatches.

SAGE assigns these tasks to different frequency representations.

Low-frequency affine alignment (LFA) predicts a six-parameter affine transformation from coarse image information and source guidance. This establishes global correspondence, and the resulting transformation also compensates the high-frequency coefficients.

High-frequency correlation refinement (HCR) then examines local correlations in the globally compensated detail representations. It predicts residual displacement, addressing spatial mismatches that a global affine transformation cannot fully resolve.

HCR also constructs a structural reliability map. The predicted displacement is multiplied by that map before warping, limiting corrections in regions where the structural evidence is less reliable.

This reliability map is an internal control over deformation magnitude, not a statistically calibrated probability that an alignment is correct.

The frequency-branch comparison in Section 4.5 directly tests this specialization:

The combined configuration produces the strongest results on both reported measures. Its high-frequency improvement of 0.317 substantially exceeds the 0.121 achieved by applying HCR to both branches.

This experiment tests the allocation of alignment mechanisms, rather than overall superiority against external systems. It supports the narrower conclusion that coarse global estimation followed by detailed local correction works better than either uniform branch assignment under the tested configuration.

After alignment, guided subband fusion (GSF) uses separate low- and high-frequency gates to determine how much information each modality contributes. The high-frequency gate additionally incorporates local correlation and reliability evidence.

Cross-frequency interaction refines the fused coefficients before inverse wavelet reconstruction produces the final image. Alignment quality therefore influences the fusion decision itself, rather than merely supplying a preprocessed input.

Removing coordination degrades structural accuracy

The paper’s MFNet ablations test whether the coordinated architecture matters beyond its complete benchmark score.

Each experiment removes one component while comparing the resulting system with complete SAGE.

Source: Section 4.3, Table 2. Lower values indicate better structural correspondence.

Removing any of the four components worsens both structural-distance measures. Removing HCR produces the largest HD95 and ASSD values among these ablations, while removing SAFE also produces a clear deterioration.

The guidance-routing experiment offers additional mechanism evidence. Sending guidance only to the low-frequency stage can impair high-frequency alignment, whereas routing guidance across all stages produces the strongest positive high-frequency response while preserving low-frequency correspondence near the no-guidance reference.

These tests support the contribution of coordinated information flow. They do not isolate every interaction among modules or establish that the same design will be optimal for unrelated sensor architectures.

The strongest benchmark results still contain meaningful exceptions

Across DroneVehicle, MFNet, and RoadScene, SAGE leads eight competing methods on three principal measures:

Source: Section 4.2, Table 1. Higher AG and SF indicate stronger image detail measures; lower HD95 indicates better performance on the reported high-percentile boundary-distance measure.

The consistency across these datasets is meaningful because their alignment conditions differ. DroneVehicle contains inherent aerial misregistration; MFNet uses synthetic translation and elastic deformation; RoadScene uses controlled horizontal displacement.

Yet other metrics reveal different outcomes. On DroneVehicle, ReCoNet obtains SCD of 1.48 against SAGE’s 1.13. On MFNet, ReCoNet again leads SCD, at 1.53 against 1.28.

More consequential for a geometry-sensitive application, SAGE’s RoadScene ASSD is 14.28, compared with MURF’s 8.43. HD95 and ASSD emphasize different aspects of boundary discrepancy, so leading one does not guarantee leadership on the other.

This is why visual sharpness, structural alignment, and downstream detection must remain separate evaluation dimensions. The reported detection advantage is specific to DroneVehicle and YOLO11m; the other two datasets do not have equivalent downstream validation in this study.

Deployment decisions depend on accuracy and processing budget

For an aerial monitoring team, Cognaptus infers that a coordinated alignment-and-fusion architecture is worth investigating when sensor displacement materially affects detection quality.

The relevant decision is not whether SAGE produces the most attractive image. It is whether its improvements in object detection and geometric consistency justify its computational requirements under the actual camera arrangement and operating conditions.

SAGE reports 91.45 GFLOPs, 3.53 million parameters, and 56.50 FPS. This is faster than several registration-intensive alternatives, but AU-Net reaches 60.93 FPS with just 14.04 GFLOPs in the reported comparison.

A deployment evaluation should therefore measure detection precision and recall, HD95 and ASSD, throughput, and latency on the intended sensor streams and hardware. Different applications may place different weights on average boundary error, occasional larger displacement, and detection performance.

The research leaves several boundaries unresolved: the main benchmark tables lack detailed image-pair counts and statistical uncertainty estimates; downstream detection is tested on one dataset and one detector architecture; and no SAGE-specific independent reproduction package is identified in the supplied research record. The reported throughput cannot establish equivalent performance on embedded hardware or different image resolutions.

Better fusion starts with knowing what should align

SAGE’s strongest contribution is architectural coordination. It preserves source-specific structural information during appearance equalization, assigns global and local alignment to different frequency components, and carries correspondence reliability into the final fusion decision.

Its benchmarks, ablations, and downstream detection experiment offer mutually reinforcing evidence for that design under the tested conditions. The metric exceptions and computational comparisons prevent a universal ranking.

For multisensor perception products, the transferable principle is to preserve and reuse information about each source throughout the processing pipeline, while measuring success against the task performed on the fused image. A visually sharp output is valuable only to the extent that it preserves the correspondence and information the application needs.

Cognaptus: Automate the Present, Incubate the Future.