A new arXiv preprint reports strong benchmark results for GAAT, a multimodal system designed to combine paired RGB and infrared UAV images after they have been only coarsely aligned. Across segmentation, object detection, change detection, tracking, 3D reconstruction and acquisition-state prediction, different GAAT configurations were reported among the top results or close to a leading score.
The study asks whether an alignment-first approach can improve perception when the two views still have small local mismatches after coarse alignment. In practical terms, the same scene may not line up perfectly in every part of the RGB and infrared images. GAAT estimates which local regions are more reliable before combining information from the two views.
A model built around imperfect alignment
The main dataset, UAVMeta, contains 2,575 synchronized image pairs from daytime and nighttime acquisitions. The data are divided into 1,715 training pairs, 572 validation pairs and 288 test pairs. Before modeling, the researchers fit a global affine transform to image matches with RANSAC, disabled homography to avoid overfitting, and warped the RGB images into the infrared coordinate system without local deformation.
GAAT estimates transformation-consistent local reliability before interaction with its syncPATC component. It then uses reliability-ranked neighborhoods for sparse bidirectional fusion and cross-granularity contrastive supervision. Put simply, the design ranks local areas according to how consistent they are with the estimated geometry, then uses those rankings when features are shared and when the model is trained.
Scores varied by task
For semantic segmentation on KUST4K, GAAT(C) obtained 82.14 on mIoU and 92.41 on mAcc, reported as the highest values in the table. On UAVMeta itself, the same configuration reported 67.79 mIoU, with a standard deviation of 1.18; 78.58 FWIoU, with a standard deviation of 1.32; and 87.44 pixel accuracy, with a standard deviation of 0.93.
The detection results followed a similar pattern. On DroneVehicle, GAAT(D) achieved 56.59 mAP, 80.12 mAP50 and 67.16 mAP75, and was reported as higher than DDQ-DETR on all three measures. On UAVMeta, GAAT(D) reported 44.86 mAP, with a standard deviation of 3.76; 62.64 mAP50, with a standard deviation of 4.40; and 52.00 mAP75, with a standard deviation of 5.13.
For scene classification on UAVMeta, GAAT(G) reported an accuracy of 63.66, balanced accuracy of 51.97 and macro-F1 of 48.12. On change detection, GAAT(CF) reported an F1 of 97.85 and IoU of 95.79 on CDD, while GAAT(CX) reported an F1 of 95.96 and IoU of 92.47 on LEVIR-CD.
The paper also reports gains across tracking, reconstruction and state prediction, although the strongest configuration varied by task or metric. On M3OT, GAAT(B) had the highest pooled HOTA, IDF1 and MOTA on Drone 1, and the highest pooled HOTA and MOTA on Drone 2. For 3D reconstruction on TSDN, GAAT plus ThermalGS reported 26.92 PSNR, 0.89 SSIM and 0.11 LPIPS. On StateBench, GAAT reached an aggregate MSPA-4D score of 52.69, close to SkySense-S2 at 52.26, and recorded the lowest listed FMCS mean absolute error at 12.66.
Promising results, narrow evidence
An ablation study reported the full configuration as strongest overall across representative tasks, while removing core modules weakened transfer on several benchmarks. That pattern supports a joint contribution from the alignment modules, but it does not isolate the effect of any one module.
The evidence has a clear boundary. Validation is centered on paired RGB-IR data and reliability supervised by affine alignment, so the study does not establish how the approach performs with other modalities or with substantially more complex geometric variation. Whether the reported transfer holds under those broader conditions remains open.
Paper data and sources
Original title: GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
Authors: Jingpu Yang, Debin Tang, Yilin Sun et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text