A preprint describes an AI system built to do two jobs in sign-language video at once: recognize a continuous sequence of signs and spot where individual signs occur. The authors call the framework SMART and report it as the best-performing system across the four listed recognition benchmarks.
The recognition measure is word error rate, or WER, where lower is better. SMART’s reported development and test WERs were 17.58 and 19.50 on PHOENIX14-T, 26.50 and 26.20 on CSL-Daily, 0.64 and 0.48 on Large-scale KSL, and 27.27 and 23.92 on DS KSL.
What was tested
The evaluation used four offline sign-language video benchmarks: PHOENIX14-T, with 8,257 videos and nine signers; CSL-Daily, with 20,654 videos and 10 signers; Large-scale KSL, with 35,987 videos and 18 signers; and the Disaster and Safety KSL dataset, with 28,250 videos and 20 signers. The two KSL datasets include temporal boundary annotations, allowing the researchers to evaluate spotting as well as recognition.
Spotting was scored with F1, a measure that combines precision and recall, at temporal-overlap thresholds of 0.10, 0.25 and 0.50. The reported results highlighted the strictest of those thresholds, known as F1@50.
How SMART links the tasks
SMART adds language-like descriptions of movement to the visual information in the video. It uses LLaVA-OneVision-7B to generate descriptions for individual frames, then combines that model-guided video-text alignment with a visual encoder enhanced by MSTA and a CSFormer module designed for spotting that is aware of continuous sign recognition.
The recognition stream starts with a CLIP ViT-B/16 encoder fitted with spatial adapters, prefix embeddings and MSTA. Its output passes through a one-dimensional convolutional network, a two-layer bidirectional LSTM and a NormLinear classifier before a CTC decoder produces the gloss sequence. The system also uses text embeddings extracted in advance with BERT for semantic supervision.
For the video-text link, the paper selected SigLIP for small-batch training. Its sigmoid loss treats each video-text pair independently instead of depending on negative examples elsewhere in the same batch. CSFormer keeps the visual and recognition streams separate while applying layer-wise bidirectional cross-attention, then combines recognition and spotting probabilities at inference with a fusion weight of 0.7 before CTC beam-search decoding.
Spotting scores were concentrated on the Korean datasets
On Large-scale KSL, SMART reported an F1@50 of 96.72 alongside a WER of 0.48. On the Disaster and Safety KSL dataset, it reported an F1@50 of 59.77 and a WER of 22.93.
The comparison with a recognition-only version of SMART showed a large difference in the reported spotting scores. On the test sets, F1@50 was 96.72 for the full system and 7.37 for SMART CSLR-Only on Large-scale KSL; on DS KSL, the figures were 59.77 and 5.90. The DS KSL test WERs were 22.93 for the full system and 23.92 for the recognition-only version.
Component tests on PHOENIX14-T produced smaller gaps in recognition. The baseline recorded development and test WERs of 18.60 and 19.80. The MSTA-only comparison recorded a development WER of 17.88, while the alignment-only comparison recorded a test WER of 19.75. The combined configuration had the best reported result in that comparison, at 17.58 development WER and 19.50 test WER.
A separate alignment comparison reported development and test WERs of 17.98 and 19.75 with SigLIP, compared with 18.62 and 20.76 with InfoNCE. The no-alignment setting was 18.60 and 19.80. With a batch size of eight, the reported scores were 17.58 for development and 19.50 for test.
A benchmark result, not a deployment test
The document is an arXiv preprint, version 1, dated 26 Aug 2026. Its evidence comes from offline computational comparisons on the four named benchmarks and their reported splits. The supplied analysis reports no study involving human participants or deployment in real-world conditions.
The results do not include inferential statistical tests, confidence intervals, error bars or run-to-run variability. The spotting evaluation is limited to the KSL datasets because the other two benchmarks do not provide the stated temporal boundary annotations. The quality and effect of errors in the generated motion descriptions were not separately validated.
The study was supported by the Institute of Information and Communications Technology Planning and Evaluation through the ITRC and ICAN programs. Each program is listed as providing 50% funding from the Korean government’s Ministry of Science and ICT.
Paper data and sources
Original title: SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting
Authors: Eunjee Choi, JungHoon Sung, Seongwhan Cho et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text