A new preprint reports that a video AI method can retain near-full benchmark performance after keeping only 10% of the visual tokens fed into the model. The method, called Token-Budget Distillation, was the strongest compressed version tested on the LLaVA-Video backbone, scoring 58.9 at the 10% retention setting and reaching 97.0% of the corresponding uncompressed model's average score.
The study addresses a practical problem in video-language models: reducing visual tokens while preserving semantic reasoning. Reducing the number of tokens can also discard information needed for reasoning, which is why the researchers tested whether a compressed student could retain the behavior of a full-token model.
Teaching a compressed model to follow the full one
Token-Budget Distillation, or TBD, uses two forward paths from the same pretrained backbone. One is a full-token teacher. The other is a compressed student that works with the reduced token sequence. The pretrained backbone is frozen, and only LoRA adapters are trained.
The student is trained with the task loss as well as several forms of guidance from the teacher. These include answer-region KL distillation, which compares the models' output probability distributions around the answer, and a ground-truth-anchored margin loss, which compares the score for the correct answer with the score of the strongest wrong answer. A reliability-aware control is also used to regulate the distillation signal during training.
Training used the LLaVA-Video-178K instruction-tuning dataset. The evaluation covered three video-language backbones, four video-understanding benchmarks, and two token-retention settings: 20% and 10%. The comparisons included the uncompressed Vanilla model, token-reduction baselines, and versions of TBD with individual components removed.
Results varied by backbone, but the compressed student stayed close
On LLaVA-Video, TBD led the compressed variants at both tested settings. With 20% of visual tokens retained, it had an average score of 59.7 and a relative accuracy of 98.4%. At 10% retention, its average score was 58.9, with relative accuracy of 97.0%. Relative accuracy is the compressed model's average score expressed against the matching Vanilla model.
The reported results were also close to the Vanilla reference on LLaVA-OneVision. At 20% retention, TBD reported an average score of 59.0, or 101.0% relative accuracy. At 10% retention, it reported 58.4 and 100.0% relative accuracy.
On Qwen3-VL-8B-Instruct, TBD scored higher than the FlashVID compression-only baseline in the reported averages. TBD scored 61.6 versus 60.9 at 20% retention, and 59.4 versus 58.8 at 10%. At the lower retention setting, the relative accuracies were 92.2% for TBD and 91.3% for FlashVID.
The extra guidance tracked the teacher's decisions
The component tests reported a stepwise increase in the average score as the listed guidance components were included. At 10% retention on LLaVA-Video, the LoRA-only version scored 58.3. The version with answer-region KL scored 58.6, the version with added ground-truth-anchored margin scored 58.8, and the full TBD configuration reached 58.9, with 97.0% relative accuracy.
The reported averages varied by compression front end. At 10% retention, TBD paired with FlashVID scored 58.9, compared with 57.7 for FastVID, 57.1 for VisionZip, and 55.0 for FastV. Their relative accuracies were 97.0%, 95.1%, 94.1%, and 90.6%, respectively.
The researchers also examined whether the student's answer preferences resembled the teacher's. In 100 randomly sampled VideoMME instances at 10% retention, the distilled student was closer to the teacher on 77 samples, compared with 23 for the compression-only baseline. The mean KL divergence, a measure of the difference between two output probability distributions, was 0.1172 for the student and 0.2310 for the baseline.
A separate analysis looked at the gap between the correct answer and the strongest incorrect answer. In 200 randomly sampled VideoMME instances, the student's mean margin was 2.7832, versus 2.2325 for the baseline. Its negative-margin ratio was 0.2400, compared with 0.3150 for the baseline. A negative margin means the highest-scoring wrong answer has overtaken the correct one.
A qualitative visualization offered a similar, but narrower, signal. In one representative VideoMME example at 10% retention, the student's attention map remained highly consistent with the teacher's and focused on the same salient regions, despite 90% of the original tokens being discarded. Because this was a single illustrative example, it does not establish how often the pattern occurs across videos or tasks.
A useful result with a costly training requirement
The method's efficiency claim has an important qualification. Adaptation still requires executing the full-token teacher, which keeps training-time memory demands high. The student also cannot recover information that the chosen training-free compression module has irreversibly removed, so the compression step constrains its possible performance.
The reported comparisons do not include confidence intervals, inferential tests, or formal uncertainty estimates. The logit analysis used 100 randomly sampled VideoMME instances, the margin analysis used 200, and the attention result came from a representative example. The findings therefore show the reported behavior on the tested backbones, benchmarks, retention settings, and sampled analyses, rather than a guarantee of general performance.
The document is marked as arXiv:2608.28138v1, dated 28 August 2026, and is described in the supplied metadata as a preprint. Its front matter also presents an ACM MM '26 proceedings citation and a DOI, but the relationship between those publication states is not clarified in the supplied material.
Paper data and sources
Original title: Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Authors: Xiaoyang Guo, Guoping Luo, Jusheng Zhang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text