A fairness-focused way of adapting CLIP as it classifies each image kept average accuracy close to the model's zero-shot baseline while reporting better subgroup measures in benchmark tests. The main experiments used CLIP's ViT-L/14 variant, with about 428 million parameters, according to an arXiv preprint.
On the paper's reported scale, zero-shot CLIP scored 86.5 for overall accuracy and 64.6 for worst-group accuracy, the accuracy of its weakest-performing subgroup. FAIRTPT scored 86.3 and 66.5, while FAIRTPT (MO), its multi-objective version, scored 86.2 and 66.6. Bias, defined as overall accuracy minus worst-group accuracy, fell from 21.9 to 19.8 and 19.6, while equalized-odds difference fell from 16.2 to 15.3 and 14.8.
For each run, the authors uniformly sampled 1,000 images from each of FairFace, CelebA, WaterBirds and UTKFace. Results were averaged over five independent random-seed runs; no significance tests, confidence intervals or variance estimates were reported.
Why ordinary adaptation can backfire
Test-time adaptation changes a model during inference, using one incoming test image before making its final prediction. In this study, each image was handled separately, without using its true target label or the sensitive attribute's label.
Some dataset-attribute cases gained more than 2.0 percentage points in overall accuracy and others lost more than 2.0 points, while aggregate accuracy remained close to or marginally below zero-shot. In most tested configurations, aggregate worst-group accuracy, bias and equalized-odds difference deteriorated by more than 2.0 points.
FAIRTPT uses a two-part objective: it minimizes entropy, or uncertainty, for the target task while maximizing marginal entropy for a specified sensitive attribute. Its entropic learning-rate adaptation control monitors the absolute change in target loss and rescales the learning rate, with the stated aim of stabilizing updates and preventing divergence.
A narrower claim about robustness
The paper reports that both FAIRTPT versions retained or improved accuracy and subgroup measures across a broad range of fairness-weight settings, with sufficiently large weights reliably balancing the objectives across datasets. Removing the learning-rate control led to notable declines in accuracy and subgroup metrics on average and across most individual datasets.
In an additional output check, average sensitive-attribute indifference rose from 36.6 before the update to 52.5 with FAIRTPT and 56.5 with FAIRTPT (MO). Average target confidence shifted more modestly, from 58.1 to 56.4 and 56.3, respectively.
What the benchmarks leave unresolved
The method depends on specifying the sensitive attribute and its possible values correctly; the authors say it is not effective when that attribute is misspecified. FAIRTPT targets the named attribute but does not prevent reliance on other, unmodeled proxy attributes.
The added adaptation also carries a practical cost. On the reported Nvidia H100 FairFace setup, processing one image took 0.43 seconds with TPT, 0.04 seconds with ZERO, 1.03 seconds with FAIRTPT and 1.71 seconds with FAIRTPT (MO).
The evidence is limited to computational experiments on four image benchmarks and specified target-spurious attribute pairs, with an additional check using CLIP ViT-B/32. The study does not describe online or continual adaptation, real-world deployment or direct human outcomes, so the results do not establish a general guarantee of fairness in use.
The document is arXiv preprint version 1, dated 26 August 2026, and no journal is listed.
Paper data and sources
Original title: Fairness-Aware Test-Time Prompt Tuning
Authors: Yoann Launay, Parameswaran Kamalaruban, Tom Kempton et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text