A mixed benchmark picture
A skeleton-based AI model reported better action-localization scores than several comparison methods on one benchmark and a mixed result on another, while using only video-level action labels during pretraining, according to an arXiv preprint. It scored 73.3% average precision on FDD, against listed comparison scores of 61.6%, 36.0% and 45.0%. On UCF101-24, its 34.1% score was above one listed method at 17.7% but below another at 35.0%.
“Zero-shot” here refers to the paper’s attempt to localize an action in space and time when pretraining did not use instance-level action labels—the person-by-person labels that say which individual performed it. The system works from detected human skeleton or keypoint sequences and is trained with video-level labels instead. Those labels identify the action attached to a clip, while the method’s task at inference is to score individual instances against a requested target action.
How the system works
During pretraining, the proposed Skeleton-Language feature Pooling Switching module, or SLPS, aggregates instance features into a video-level feature and applies contrastive learning with text embeddings for the action labels. At inference, a user supplies a target-action prompt; the system compares each instance’s feature with the prompt’s text embedding and uses the similarity to assess each instance. The SM-DCL component merges instances from different scenes into a shared context and applies contrastive learning to distinguish actions across instances.
The study used Kinetics-400 for pretraining, then evaluated localization or classification on UCF101-24, FDD, RWF-2000 and MF. On the two classification datasets, the proposed row reported 84.0% accuracy on RWF-2000 and 92.7% on MF. Those numbers exceeded PointNet++ at 78.2% and 89.2% and DGCNN at 80.6% and 91.3%, respectively, but remained below SPIL at 89.3% and 98.5%.
Speed and comparison results
Against appearance-based methods on RWF-2000, the model reported 84.0% accuracy, compared with 69.3% for X-CLIP, 71.3% for EVA, 72.8% for ViFi-CLIP and 80.0% for Qwen2.5-VL-7B-Instruct. The paper also reports about 70 million model parameters, approximately 100 times fewer than recent large vision-language models. In a throughput comparison, it processed about 1,900 frames per second versus about eight for Qwen2.5-VL-7B-Instruct, described in the paper as roughly a 240-fold speedup.
The paper also compared component and data-setting variants in ablations. With SM-DCL, AP was 34.1%, compared with 31.2% without it. A three-shot ablation reported 33.9% AP, close to 34.1% for the version with SM-DCL. These are within-experiment variant comparisons, so they describe an association rather than a randomized estimate of what caused the difference.
What remains uncertain
The authors report higher robustness than appearance-based approaches when inputs are blurred or when frames are dropped through masking. However, the supplied analysis says the exact plotted values for that degradation test were not tabulated. It also flags a tension in the pretraining assumption: approximately 20% of Kinetics-400 videos reportedly contain scenes with multiple people performing distinct actions, even though the setup treats a video as having one action.
Taken together, these results should be read as benchmark comparisons, not a causal test of whether the architecture or SM-DCL produced the gains. The supplied analysis reports no confidence intervals, p-values or hypothesis tests for the comparisons. It remains open how well the method generalizes beyond the named action datasets and selected degradation conditions. The work is an arXiv preprint, version 1, dated 26 Aug 2026.
Paper data and sources
Original title: Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining
Authors: Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text