A proposed training method for moving knowledge between AI models built from different kinds of data performed best in seven of eight reported classification transfer settings. Its one miss was the CrisisMMD text-to-image direction, where it ranked second. The result comes from an arXiv preprint examining whether a student can learn intermediate knowledge from a teacher whose feature structure is different, without aligning their internal units directly.
A different kind of bridge
The framework is built around a codebook, a learned set of teacher representations that gives the student a common interface. It creates a vector-quantized teacher codebook, scores codes by task relevance and compatibility with the student, selects high-importance codes for student-side quantization, and adds codebook-induced prediction matching. The training runs in three stages. The intended effect is to transfer useful teacher cues without asking the student to reproduce an internal layout it does not share.
To test it, the authors used four multimodal classification benchmarks, dividing each into training, validation and test portions in a 6:2:2 ratio. They selected the best validation checkpoint and reported test accuracy averaged over five independent runs. The second part of the evaluation used NYU-Depth V2 for semantic segmentation, which labels pixels across 40 classes. Those results also used the 6:2:2 split and five-run averages, measured by overall accuracy, class-averaged accuracy and mean intersection-over-union, a measure of overlap between predicted and true regions.
Strong results, with a clear exception
On classification, the advantage was broad rather than universal. The method led in seven of the eight teacher-to-student directions, but it did not take the top spot in CrisisMMD text-to-image. That makes the finding a strong result across the reported tests, not a clean sweep.
Segmentation produced a similar pattern with a sharper qualification. For depth-to-RGB transfer, the reported scores were 0.5632 for overall accuracy, 0.2909 for class-averaged accuracy and 0.2003 for mean intersection-over-union. For RGB-to-depth, they were 0.5218, 0.2184 and 0.1456, respectively. The method was best on most of the listed measures, but RGB-to-depth overall accuracy was slightly below the no-distillation baseline, even though class-averaged accuracy and mean intersection-over-union were higher.
To check whether the classification gap might reflect randomness, a controlled analysis held the data split and teacher checkpoint fixed, varied the remaining training randomness across ten runs, and used two-sided paired t-tests. The method's improvement was statistically significant at the study's 0.05 threshold in five of eight directions. Two further settings were near that threshold, and the paper describes the CrisisMMD text-to-image evidence as limited.
The ablation tests offered a clue about what mattered inside the system. The strongest overall result came from combining the student-side codebook objective with codebook-induced prediction matching. Matching the teacher prediction after codebook transformation outperformed matching the original teacher prediction across all evaluated datasets and transfer directions.
The bill comes at training time
The main practical tradeoff appears during training. In the reported VGG-Sound comparison, the three-stage method had the longest wall-clock training time. Leave-one-code-out scoring in Stage 2 accounted for 7.9% of total time in video-to-audio transfer and 7.7% in audio-to-video transfer under the default configuration. At inference, the added teacher, codebook, projection and codebook branch are removed, so the paper reports no extra inference parameters, memory or computation relative to the student-only model.
Another limitation is the amount of teacher information retained. The method uses a fixed default of 50% for classification and 12.5% for segmentation. Those ratios were selected coarsely rather than optimized for each setting, so the experiments do not show that either choice is generally best.
The MM-IMDB test also has a narrower scope than a standard benchmark result might suggest. It used a customized single-label subset of 5,778 paired poster-text samples spanning eight classes, and the official split was not used because filtering changed the benchmark. The result therefore applies to that defined setup, not directly to the standard multi-label protocol.
Taken together, the reported experiments point to codebook-level interfaces as a workable way to move teacher signals across structurally mismatched feature spaces in the tested tasks. The approach led most comparisons and added no reported burden at inference, but it demanded a longer training process and left the best retention policy unresolved.
Paper data and sources
Original title: Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features
Authors: Dae Ung Jo, Jongin Lim, YoungJoon Yoo, Daeho Um
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text