An artificial intelligence tool described in a new arXiv preprint can turn video-based object masks into labels for four-dimensional LiDAR data without manually annotated 3D data. Called LiDAR-SAM2, it was tested on the SemanticKITTI benchmark and its labels were used to train segmentation models. The resulting scores beat those from point labels and SAM2 labels alone, but stayed below scores from full ground-truth labels. The document is an arXiv preprint, version 1, dated 26 Aug 2026.
How the labels are built
The paper tackles whether a 2D video foundation model can supply the supervision needed for high-quality 4D LiDAR training data, removing the need for human LiDAR labeling. In the reported setup, 64-beam LiDAR sequences were synchronized with two forward-facing RGB cameras. SAM2 was applied to the camera streams, its masks were propagated over time, and the masks were transferred to LiDAR points through geometric projection. The pipeline then produced point-wise 4D labels without manually annotated 3D data.
LiDAR-SAM2 also merges information across views and time. Spatial aggregation combines view-specific masks that share LiDAR points. Temporal aggregation propagates earlier masks using vehicle ego-motion, then fuses them with current masks through voxel-wise majority voting of mask IDs. In the reported comparison, aggregation increased the average number of segments from 22.12 to 73.84. Recall rose from 13.59% to 79.58%, while precision fell from 91.90% to 85.93%, a trade-off between recovering more segments and retaining a higher fraction of precise ones.
The system's training was staged. First, it aligned camera and LiDAR information at the frame level. Then it learned temporal propagation and object-level consistency for 4D segmentation. The paper describes LiDAR-SAM2 as adapting SAM2's video-segmentation kernel to spatio-temporal LiDAR through a tailored interface and two-stage objective. It reports that one click per object produced a consistent mask track.
For the labeling-tool evaluation, the researchers simulated interaction on four-frame sweeps. Each sweep began with a positive click at an object's center, followed by corrective clicks in the larger error region. The simulation averaged roughly ten prompts per frame. It used 0.2-meter voxelization and a carryover rule that reused previous masks when their overlap with ground truth exceeded 0.8 IoU.
Where the scores landed
The main semantic test asked how well each point received the right class. The study used mean intersection-over-union, or mIoU, an average overlap score across 19 SemanticKITTI classes. For MinkUNet, the scores were 5.2 with point labels, 24.3 with SAM2 labels, 55.2 with the paper's labels, and 63.8 with full ground truth. For PTv2, they were 7.4, 26.4, 61.3, and 70.3 in the same order.
The same pattern appeared in 4D panoptic segmentation, which also evaluates whether object identities remain associated over time. The study used LSTQ, alongside association and classification scores and separate IoU measures for stuff and thing classes. On LSTQ, 4D-PLS scored 46.9 with the paper's labels versus 62.7 with full ground truth. 4D-StOP scored 57.7 versus 67.0, and Mask4Former scored 64.8 versus 70.5. The paper reports that its labels were below full ground truth across the listed panoptic measures.
What the benchmark leaves open
The evidence is a SemanticKITTI benchmark comparison, with the training data used for training and the validation data used for the reported results. The simulated protocol also used ground-truth overlap in its carryover rule, making that condition part of the reported setting. The manuscript refers to supplementary material for additional aggregation details, but those details are not included in the supplied text.
The authors interpret the results as evidence that automatically curated SAM2 supervision can make interactive 4D LiDAR labeling scalable, reduce annotation burden, and support strong semantic and panoptic performance with minimal prompting and no manual LiDAR labels. The benchmark supports a more measured conclusion: the generated labels were useful enough to train models that outperformed the point and SAM2-label comparisons, while still falling short of full ground truth.
The work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea under grant NRF-2025S1A5C3A04022639.
Paper data and sources
Original title: Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models
Authors: Jihun Kim, Hyun-Kurl Jang, Hyemin Yang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text