In tests of three language-model backbones, Acmite's reported scores moved in favorable directions on several gender-bias benchmarks, while its average scores on general-capability tasks stayed close to those of the unmodified models, according to an arXiv preprint. The study reports descriptive comparisons only; it gives no confidence intervals, significance tests or other uncertainty estimates.
Phi-4-mini-Instruct, a 3.8B model, supplied the main experiment. Qwen3-4B-Instruct-2507 and Llama-3.1-8B-Instruct, with 4B and 8B parameters, were used to examine whether the pattern appeared across different model backbones.
A three-part intervention
The method has three linked parts: stereotype concept retrieval, concept-guided adapter optimization and similarity-based inference routing. It first gathers concepts related to stereotypes, adjusts an adapter using those concepts, then decides at inference time whether an input is similar enough to activate that adapter.
Its setup used Qwen3-Embedding-0.6B for retrieval, 3,953 stereotype concepts and 3,704 training contexts. It used lightweight LoRA adapters and set a global 0.60 threshold for inference routing.
The comparison included the untouched backbone and five named debiasing baselines: BiasDPO, SynDebias, GenderAlign, BiasEdit and DSO. The methods were evaluated on identical benchmark splits and metrics.
The bias tests used gender-specific subsets of BBQ, CrowS-Pairs and StereoSet. BBQ includes ambiguous and disambiguated question answering, while CrowS-Pairs and StereoSet use model likelihoods. The reported conventions treat a BBQ bias score of zero as ideal, stereotype scores of 50 on CrowS-Pairs and StereoSet as ideal, and higher accuracy, language-model scores and ICAT as better.
Several measures moved, but not uniformly
On Phi-4-mini, Acmite's two BBQ bias scores were -0.46 and 0.58, with disambiguated accuracy of 90.09. Its CrowS-Pairs stereotype score was 55.63, compared with 59.06 for Base, and its StereoSet stereotype score and ICAT were 64.39 and 66.54. Against the benchmark's preferred directions, those results put Acmite closer to the BBQ and CrowS-Pairs ideals than Base and gave it the best reported StereoSet ICAT.
Qwen3 showed a mixed picture. Acmite's ambiguous BBQ bias score was 0.00, versus 0.14 for Base; its CrowS-Pairs stereotype score was 56.25 versus 57.50; and its StereoSet stereotype score was 67.02 versus 67.54. ICAT was 61.30 for Acmite and 60.50 for Base. That meant a zero ambiguous BBQ bias score and a higher ICAT, while the other likelihood metrics changed modestly.
On Llama-3.1-8B, Acmite's two BBQ bias scores were 0.71 and 2.59, with disambiguated accuracy of 82.55. Its CrowS-Pairs stereotype score was 59.38, compared with 63.44 for Base, and its StereoSet ICAT was 60.03. In the reported comparison, the BBQ scores were nearer zero, the CrowS-Pairs score was lower than Base, and ICAT was highest among the compared rows.
Capability stayed close on average
The reported three-task averages were close to the base models. Acmite's average was 73.24 for Phi-4-mini, compared with 73.10 for Base; 68.08 for Qwen3, compared with 68.21; and 74.29 for Llama-3.1-8B, compared with 75.54. The analysis describes these as benchmark-specific, descriptive capability comparisons.
At task level, the reported Acmite scores for ARC-Challenge, GSM8K and PIQA were 61.60, 81.20 and 76.93 for Phi-4-mini, and 55.29, 79.53 and 69.42 for Qwen3. The mixed task-level changes show why near-identical averages do not mean identical behavior on every test.
An ablation test on Phi-4-mini compared the full objective with versions missing each of its two optimization terms. The full setup had a reported mean change of 22.9% across eight metrics, compared with 18.3% when one term was removed and 2.4% when the other was removed. In the full setup, the two BBQ bias-score changes were 71.1% and 50.8%, while disambiguated accuracy changed 0.7% and ICAT 8.8%.
Routing also changed which inputs received the adapter. In the Phi-4-mini capability test, ARC-Challenge, GSM8K and PIQA scores were 57.51, 77.71 and 69.21 in the always-on comparison, and 61.60, 81.20 and 76.93 with retrieval-guided routing. The adapter was activated for 0.96% of ARC-Challenge inputs, 0.96% of GSM8K inputs and 9.25% of PIQA inputs.
The threshold comparison described a relevance-coverage trade-off: lowering the threshold activated many more inputs, but did not produce stronger reported results on every metric. No formal threshold-selection analysis or uncertainty estimate was reported.
What the results do not establish
In an illustrative founder scenario, Base assigned product vision to Leah and business operations to Mark even though the two had equal contributions. Acmite instead attributed both vision and execution jointly to the two founders. The case is illustrative, not a systematic estimate of how often models make such assignments.
The paper's scope is narrow. It evaluates gender bias with binary gender representation and does not establish applicability to race, religion, age, intersections or non-binary identities. The benchmark scores and selected qualitative cases are proxies for bias and capability, rather than direct measures of human discrimination or deployment outcomes.
Taken together, the preprint reports a benchmark pattern in which bias scores moved favorably on several measures while general-capability averages remained near base levels. The supplied evidence does not establish causal effects, statistical significance or fairness gains in real-world use.
Paper data and sources
Original title: Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Information
Authors: Tian Lan, Xiaoqing Cheng, Han Zhang, Jiang Li
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-10-01
DOI: Not available
Original paper · Full text