The strongest result comes with a warning
An arXiv preprint reports a split result for an AI explanation method: in tests of two four-option classifiers, zeroing the embedding of the token ranked highest for a selected answer’s width changed that width more than zeroing randomly chosen eligible tokens did. But numerical checks later found that a stable token ranking did not always mean the attribution values were accurate.
The method focused on a quantity called credal width. For a selected answer, it is the gap between plausibility and belief. In practical terms, it is the mass assigned to sets that contain that answer along with at least one other option. That is different from the pignistic probability used to choose an answer, which shares each set’s mass among its members before adding the selected answer’s share.
What the token test actually measured
To ask which parts of a question and its answer options influenced the reported width, the study ranked input tokens by their attribution to the selected answer’s corrected width. It then zeroed individual token embeddings while preserving sequence length and the attention mask. Eligible tokens came from the question or option text and contained a letter or digit; the top-ranked token was compared with 20 uniformly sampled eligible positions.
The classifiers were adapted from SmolLM3-3B and Llama-2-7B using 10,178 MedQA US training questions, 1,272 development questions and 1,273 test questions. Prompts were truncated to 512 tokens.
On the first 100 test questions for each model, the top-ranked token produced an average absolute width-change advantage of 0.0381 for SmolLM3-3B and 0.0450 for Llama-2-7B over the random-token comparison. The 95% confidence intervals were 0.0243 to 0.0532 and 0.0295 to 0.0617, respectively, so both excluded zero.
That result shows sensitivity to the specified embedding operation, not recovery of clinical evidence. The operation changed token embeddings rather than deleting or meaning-preservingly rewording text. The evaluation used fixed 100-question MedQA panels and two four-option classifiers.
The score was not a better error signal
The same evaluation reported full-test accuracy of 49.3% for SmolLM3-3B and 45.7% for Llama-2-7B. It also tested whether width could rank wrong answers. Width-based error-detection AUROC, a measure of how well a ranking separates errors from non-errors, was 0.634 and 0.598, while ranking by low predicted-answer probability scored 0.680 and 0.671.
The model’s correction machinery added another complication. Its classifier used 14 sigmoid outputs for four singleton sets, six pairs and four triples. The conversion removed negative intermediate values, assigned any shortfall to the full set and rescaled corrected masses to sum to one. Across the reported configurations, rescaling occurred in 95.84% to 99.92% of the 1,273-question test sets. Negative intermediate values occurred in every tested prediction.
A stable ranking is not the same as an accurate score
Numerical checks on four synthetic cases exposed the difference between a ranking that looks right and values that add up accurately. Median Spearman rank correlations were 0.9951 for training references, 1.0000 for aligned-paired references and 0.9840 for zero-baseline references. But the corresponding median completeness residuals, the remaining mismatches between the attribution sum and its target, were 0.0652, 0.0144 and 0.6428. The training-reference residual exceeded the stated 0.01 tolerance, and its relative vector difference was 0.1563, above the 0.10 tolerance.
Refining the calculation helped in the aligned cases. Denser FP32 Integrated Gradients passed the reported residual and attribution-agreement checks for all four aligned references, with a maximum residual of 1.105 × 10−5. In one case, refining five of 128 training-reference integrals yielded a mean residual of 0.00401 and a maximum residual of 0.02454.
Yet the target itself changed when the reference inputs were aligned. Across the four cases, alignment shifted the width difference by 0.481233, 0.373522, 0.202545 and 0.536730. In one case, the natural difference was negative 0.478592, the aligned difference was positive 0.002640 and the signed Integrated Gradients contributions summed to positive 0.002651. The attribution therefore described the aligned difference, not the original natural difference.
A second version remains untested
The revised synthetic dataset did not lead to a reported second-round attribution study. After the nonlinear model schedule, development errors for training seeds 7, 17 and 23 were 0.4263, 0.4261 and 0.4359. No evaluated development checkpoint met all required criteria, so no V2 attribution experiment was reported.
The narrower conclusion supported by the tests is that the top-ranked token produced a larger width change under this masking test. The stronger conclusion, that the resulting scores reliably measure credal width or recover clinical evidence, is not supported by these experiments. The study did not independently validate credal width as epistemic uncertainty or establish generalization beyond the evaluated panels, models and masking operation.
The document is an arXiv version 1 preprint dated 29 September 2026.
Paper data and sources
Original title: XU-RS: Explaining Credal Width in Random-Set Language Models
Authors: David Achara, Maryam Sultana, Alexander D. Rast, Fabio Cuzzolin
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text