An approach that transfers internal features from a multimodal model into a CLIP student recorded the strongest listed average in a set of compositionality tests, which ask whether a model can handle combinations of visual details and their descriptions. The InternVL3.5-2B MLLMCLIP row scored 40.4 across 11 benchmarks, compared with 37.1 for CLIP and 38.3 for TripletCLIP. The figures are reported benchmark averages, and no inferential uncertainty is provided for the comparison.
Passing along hidden signals
The paper asks whether a generative multimodal large language model, or MLLM, can pass its multimodal and compositional knowledge directly to a discriminative CLIP student, an image-and-text model, through hidden-state features, without synthetic hard-negative data. In this setup, feature-level distillation means using the teacher's internal signals to train the student. Attention selects the teacher tokens, while a CKA-based feature-alignment loss, a way to compare the structure of internal representations, is used to handle the mismatch between the two architectures.
Training used CC3M image-caption pairs. A frozen MLLM processed the pairs offline for per-layer feature extraction before the student was trained. Evaluation covered 11 compositionality benchmarks, 13 zero-shot classification datasets and two image-text retrieval benchmarks. The paper's evidence is therefore a set of model and benchmark comparisons across three kinds of tests.
The pattern extended beyond composition
The pattern extended to classification without task-specific retraining. On the aggregate from 13 zero-shot classification datasets, the InternVL3-2B MLLMCLIP row reported 31.4, versus 21.2 for CLIP and 24.2 for TripletCLIP. No inferential uncertainty is reported for these classification scores.
Image-text retrieval pointed in the same direction in the selected MSCOCO comparison. For the InternVL3-2B MLLMCLIP row, image-to-text R@1 was 14.7 and text-to-image R@1 was 11.6, compared with 9.02 and 6.85 for CLIP. R@1 asks whether the correct match appears first in the ranked results, so these figures describe top-choice retrieval performance. The study reports these as scores, without inferential uncertainty.
Teacher choice entered the picture
Teacher choice entered the picture in a SugarCrepe pilot. Base MLLMs generally scored above embedding-tuned counterparts in the reported comparison: Qwen3-VL-2B averaged 94.5, compared with 83.0 for Qwen3-VL-Embedding-2B, while LLaVA-1.6-mistral-7B averaged 85.0, compared with 66.7 for VLM2Vec-v1. The pilot makes the teacher part of the performance comparison, rather than a background detail.
What the comparison leaves open
The evidence has clear boundaries. Configurations are reported with a single seed rather than repeated runs, and the study reports no inferential tests, confidence intervals or p-values. The main-table baselines also use different external models, although an appendix comparison uses the same Qwen3.5-2B teacher for data-level and feature-level supervision. The reported averages therefore do not show whether the same ordering would hold across independent runs or other benchmark families.
That leaves a narrower conclusion. On the listed comparisons, MLLMCLIP configurations recorded higher reported averages than the cited CLIP and TripletCLIP rows on compositionality and classification, and the InternVL3-2B row also reported higher selected MSCOCO retrieval scores than CLIP. Whether the pattern generalizes to other MLLM families, domains and tasks, and whether it holds across repeated runs, remains open.
Paper data and sources
Original title: MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Authors: Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text