A preprint reports that adding internal features from a diffusion model to CLIP-based training was associated with higher scores when image-recognition systems had to identify attribute–object combinations absent from their training examples. The DIFFCZSL method was tested on three compositional zero-shot learning benchmarks—MIT-States, UT-Zappos and C-GQA—in both closed-world and open-world evaluations.
The test focused on unfamiliar combinations
In the task, models train on some attribute–object compositions and are tested on unseen ones. The researchers swept a calibration bias to balance predictions for the two groups, then reported the area under the seen–unseen trade-off curve, or AUC, the best harmonic mean, or HM, and the maximum accuracy for seen and unseen compositions. For open-world evaluation, infeasible compositions were removed after training.
Scores were higher across the benchmark tests
On MIT-States, DIFF-Troika recorded a closed-world HM of 39.9 versus 39.2 for the corresponding baseline, while AUC was 23.1 versus 22.1. In open-world testing, HM was 20.7 versus 19.1 and AUC was 7.8 versus 6.8. DIFF-CAMS posted closed-world HM/AUC of 40.7/24.0 against 40.3/23.4, although its maximum Seen accuracy was 52.5 rather than 52.8.
On UT-Zappos, DIFF-Troika’s closed-world HM reached 57.6, compared with 54.6 for Troika, while AUC reached 46.2 versus 41.7. In open-world testing, the same comparison was 50.1 versus 47.8 for HM and 36.6 versus 33.0 for AUC; DIFF-CAMS moved from 50.8 to 52.6 in HM and from 35.7 to 38.1 in AUC.
On C-GQA, DIFF-Troika’s closed-world HM/AUC was higher at 32.0/14.3 than the baseline’s 29.4/12.4. DIFF-CAMS recorded 34.2/16.2 in closed-world evaluation and 14.5/4.3 in open-world evaluation.
The extra signal is used only while training
DIFFCZSL uses a frozen diffusion model as an auxiliary training signal. It extracts intermediate features conditioned on attribute–object text, pools them, projects them into CLIP’s visual and textual spaces, and aligns them with image and compositional-text representations through image-side and text-side distillation. The diffusion branch is removed during inference, so the reported pipeline adds no inference-time computational cost. The implementation used CLIP ViT-L/14 and CleanDIFT built on Stable Diffusion 2.1.
An ablation found higher reported results with either image-side or text-side distillation, while using both produced the highest HM and AUC on both tested datasets. In a controlled comparison with DINOv3 as the alternative teacher, diffusion supervision reached HM/AUC of 39.9/23.1 on MIT-States and 57.6/46.2 on UT-Zappos; DINOv3’s improvements were described as modest and inconsistent.
The trade-off is time and model size
For the CSP baseline, the method added 32 seconds per epoch on MIT-States, 27 seconds on UT-Zappos and 57 seconds on C-GQA—about 3.5%, 4.7% and 2.6%, respectively. Troika incurred additional training times of 6 minutes 9 seconds, 2 minutes 48 seconds and 3 minutes 31 seconds on the same datasets, while inference efficiency was unchanged.
Training also required more trainable parameters: 2.7 million extra for CSP and Troika and 17.7 million for CAMS. Total model sizes rose from 27.6 million to 30.3 million for CSP, 29.1 million to 31.8 million for Troika, and 62.1 million to 79.8 million for CAMS.
A useful benchmark result, with open questions
A stability check found only minor metric variation across three random seeds, but it covered the Troika-based model on MIT-States and UT-Zappos rather than every configuration. The evaluation was limited to the three named benchmarks, so whether the reported pattern transfers to other datasets, backbones or diffusion models remains open.
The work is an arXiv version 1 preprint dated 20 August 2026.
Paper data and sources
Original title: DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations
Authors: Hangyu Tian, Zhenqi He, Yanghao Wang, Long Chen
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text