The clearest reported difference in a new vision-language model study appeared in an early layer: Visual Heads Only had a POPE F1 score 13.7 points above All Attention Heads at layer 7. The gap diminished in middle and deeper layers.
The study asks whether aggregating attention across all heads overlooks heterogeneous head contributions and whether critical visual-token localization is concentrated in a small subset of heads. ProViP is a training-free progressive pruning framework that removes redundant visual tokens before language-model reasoning and applies head-aware pruning during reasoning.
How the comparison was made
ProViP is training-free and progressive. It removes redundant visual tokens before language-model reasoning and applies head-aware pruning during reasoning, making the choice of which visual tokens to retain part of both stages.
To define visual heads, the method extracts semantic anchor tokens, scores each head by the variance in its cross-modal attention over those anchors, and ranks the heads by confidence. In the layer-wise diagnostic, researchers selected the top six of 32 heads as visual heads and reduced 576 visual tokens to 16 at the pruning layer.
That pattern was not uniform across the model. Visual Heads Only had its clearest reported advantage in shallow layers, while the difference from All Attention Heads became smaller later. In a separate visual-head-number ablation at the same shallow pruning layer, the performance gap narrowed as more visual heads were selected.
Reported results across models
On LLaVA-1.5-7B, the ProViP configuration retained 95.9% of original performance and reported a 1.62-times inference speedup under an 88.9% pruning ratio. Across the evaluated pruning ratios, its task performance was higher than that of all listed baselines. Its margin over HoloV was 0.4% at 66.7% pruning and 1.2% at 88.9% pruning.
The reported efficiency measurements showed 126 milliseconds of latency for the configuration retaining 64 visual tokens, 38.2% below vanilla latency, while prefilling time was 62.2% lower. In a high-resolution evaluation using LLaVA-NEXT-7B, ProViP retained 320 visual tokens and reached 95.9% average task performance.
The paper also reports a separate evaluation on Qwen2.5-VL-7B. At 10% pruning, ProViP’s average score was 3.2% above PDrop; at 20% pruning, its POPE score was 87.0% and was reported above vanilla.
An ablation compared the framework’s full configuration with pruning performed only before the base language model. At the retain-192-token setting, corresponding to 66.7% pruning, full ProViP had a reported average of 99.3%, compared with 97.0% for the pre-language-model-only configuration.
What the results leave open
The evaluation covered LLaVA-1.5-7B and LLaVA-NEXT-7B, with Qwen2.5-VL-7B used for a separate generalization evaluation. The reported benchmarks were GQA, MMB, MMBCN, MME, POPE, SQA and VQAText.
The authors report that ProViP requires access to intermediate language-model results and explicit attention-score computation. Those requirements restrict direct use to open-source architectures, and the additional computation can offset some of the reported acceleration.
The work is identified as arXiv:2608.25332v1 [cs.CV], dated 26 August 2026, and its reported comparisons remain tied to the models, benchmarks, pruning settings and inference conditions tested in the preprint.
Paper data and sources
Original title: Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
Authors: Chaofang Ma, Lin Jiang, Carol Jingyi Li et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text