Strong scores with fewer visual tokens
A training-free method called DIPrune kept benchmark performance close to the unpruned level in tests that stripped away large numbers of visual tokens, the chunks of visual information passed into a multimodal language model. On LLaVA-v1.5-7B, it retained 99.9% of unpruned average performance with 192 tokens left, a 66.7% pruning rate. With 128 tokens left and 77.8% pruning, it exceeded the best baseline by 2.3%. At 64 tokens, or 88.9% pruning, it retained 97.7% of performance.
The pattern held across other model families in the reported comparisons. On LLaVA-NeXT-7B, DIPrune retained 98.2% of average performance at 88.9% compression, compared with 87.2% for FastV and 90.6% for SparseVLM. On Qwen2.5-VL-7B, its average at 88.9% pruning was 93.3%, against 90.5% for HoloV, with gains of 3.7 on MMBench and 3.5 on POPE. On dense and mixture-of-experts versions of Qwen3-VL, the largest reported margins at the same pruning ratio were 11.3 on POPE and 11.7 on MMBench, respectively.
Efficiency tests pointed in the same direction. DIPrune's measured time to first token, or TTFT, was 55.6% lower than the baseline, while ITT remained nearly identical. At comparable TTFT, its reported accuracy was 97.7%, versus 93.7% for HoloV. The pruning step added 0.24 milliseconds of overhead, equal to 0.74% of TTFT.
A score built around change, not just attention
DIPrune is designed to choose tokens for the task at hand rather than rely on a fixed, task-agnostic rule. Its derivation treats pruning as a constrained selection problem: choose a binary mask that keeps exactly K visual tokens while minimizing the change in the model's final task loss. In plain terms, the target is to discard tokens that matter least to the eventual task, subject to a fixed token budget.
Behind that objective is a split between inter-layer and intra-layer importance. The first uses per-token gradient energy, a measure of the strength of a token's gradient signal. The second uses per-token feature magnitude. The pruning rule keeps the K tokens with the largest combined score.
The implemented score combines two kinds of evidence. Semantic Alignment averages the attention that text queries assign to visual tokens, then converts each token's rank into an inverted score. That makes the score depend on a token's relative position in the ranking rather than on the absolute size of attention. Semantic Evolution looks at how a token's rank shifts between layers, clips extreme shifts and normalizes the result, emphasizing tokens whose priority is rising while suppressing fluctuations.
The two signals are fused into one DIPrune score, with weights that add up to one, and the highest-scoring tokens are kept. The design addresses what the paper calls an Early Inertial Bias: tokens that attract high attention in shallow layers can stay dominant, while emerging tokens may be removed before their deeper semantic importance becomes visible.
The default procedure prunes once at the 10th language-model layer. It compares ranks one layer apart, clips rank displacement at a threshold of 200, and uses weights of 0.6 for Semantic Alignment and 0.4 for Semantic Evolution. Ablation tests selected the 10th layer as a balance between semantic preservation and efficiency and found those weights optimal on LLaVA-NeXT-7B.
What the tests leave open
The findings come from computational evaluations across LLaVA, Video-LLaVA and Qwen-VL model families and a broad public benchmark suite. The authors say the method's margin may be smaller in highly cluttered scenes or when the text query is extremely short. They also note that latency benefits may be constrained when compression has to happen before the language model.
The manuscript is a preprint. Funding information is not reported.
Paper data and sources
Original title: DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
Authors: Shuo Yang, Changbai Li, Linlin Yang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-10-06
DOI: Not available
Original paper · Full text