Open-weight language models can be drawn toward the wrong source when textual, numerical and external-tool evidence disagree, according to an arXiv preprint. The tests found that the models did not apply one consistent rule for deciding which evidence to trust: their choices varied by model family and by the way conflicting information was presented.
The strongest problem appeared in conflicts involving a simulated tool forecast. Many models gave heavy weight to an external forecast even when measurements in the surrounding context pointed to the correct answer. In some settings, accuracy fell to near zero or below chance, a pattern consistent with systematically preferring the wrong evidence rather than simply becoming uncertain.
A deliberately controlled test
The study was designed as a methods benchmark for a narrow question: how do language models arbitrate when exactly one source matches the ground-truth label and other sources conflict with it? The researchers varied factors including whether the evidence was textual or numerical, how recently it appeared, whether its reliability was signaled, where it came from and where it was placed in the prompt.
Each test instance was built from a synthetic risk trajectory whose values ran from 0 to 1. The model had to predict whether a future value would be HIGH or LOW, using a threshold of 0.5; the forecast could look at least one step ahead. The benchmark therefore reduced the decision to a binary choice rather than asking for a full numerical forecast or a calibrated probability.
The main trajectories contained 16 time steps, with a one-minute sampling interval and a one-step forecast horizon. The labels were balanced between HIGH and LOW. The main conflict experiments used 2,000 instances in each setting, including 1,000 for each label; sensitivity analyses used 1,000 instances per setting.
The evaluation covered Qwen3 models at 1.7B, 4B, 8B and 14B, along with Gemma-2-9B-It, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3. Predictions were taken from the models' logits for the binary answer tokens, and results were averaged across three random seeds. Accuracy was also compared with reference tests in which the available source was aligned with the ground-truth answer.
Different models, different instincts
When each modality was tested on its own, all of the models reached very high accuracy. Once the sources were put into conflict, however, the families separated. Qwen3 tended to favor numerical evidence, while Llama and Mistral tended to favor text. Gemma showed the most balanced pattern between the two.
The position of a source in the prompt also mattered. Accuracy was generally higher when the source carrying the correct label appeared later. That later-position advantage was stronger for text than for numerical evidence, although numerical information remained influential even when its position changed.
The results also point to a difference between two signals a model might use to arbitrate a conflict: recency and stated reliability. Across most of the models, conflicts over reliability were linked to larger drops in performance than conflicts over temporal recency. In the benchmark, explicit reliability appeared to be a weaker cue than how recently the information had been presented.
The simulated tool conflicts stood out from the other tests. They produced the strongest degradation across the experiments, with many models relying heavily on the external forecast even when contextual measurements disagreed. That result does not show that any particular real-world tool is unreliable; it shows how the tested models behaved when the tool output was deliberately made wrong.
The effect was not uniform across the models. Qwen3 and Gemma were especially susceptible when the correct contextual evidence appeared before the incorrect tool forecast, often reaching near-zero accuracy in that arrangement. Llama and Mistral were less influenced by the incorrect forecast.
Some benchmark settings generated below-chance accuracy. In ordinary terms, the models were not merely guessing without a clear preference; their answers were aligned with the incorrect evidence often enough to perform worse than random choice. The finding is consistent with a systematic preference for incorrect evidence rather than mere uncertainty.
Small changes to the prompt could shift the result
The researchers also tested how robust the pattern was to changes in the setup. Domain changes generally had little effect in text-only settings, but changing the answer choices shifted accuracy in conflict conditions even while unimodal performance stayed stable. The 14B Qwen3 model was comparatively stable across these settings, while the wider results still showed differences by model family.
Taken together, the results point to recurring, model-specific preferences for particular cues. A model could be highly accurate when evidence was aligned and still behave predictably badly when the evidence was arranged against its preferred cue. External forecasts, prompt position, modality and reliability did not carry the same weight across the models.
A narrow stress test, not deployment guidance
The study uses synthetic data rather than real-world records, and its forecasting task is deliberately binary. Those choices make it possible to control which source is correct, but they also simplify the ambiguity and complexity of real decisions. The benchmark therefore cannot show how the same patterns would transfer to naturally occurring conflicts in real-world settings.
The tool forecasts in the experiment were simulated and intentionally incorrect, while reliability cues were explicitly signaled. The evaluation also covered only the named open-weight instruction-tuned models. It did not test closed or commercial systems, full numerical forecasting, calibrated probabilities or the option to abstain.
The reported results describe accuracy patterns, not causal proof about why a model makes a particular choice. The supplied analysis reports no hypothesis tests, p-values, confidence intervals or other inferential statistics; sensitivity estimates were summarized as mean plus or minus standard deviation. Further work would be needed to test whether the patterns persist with continuous forecasts, realistic tool outputs, broader model families and prompts or training methods designed to make reliability matter more.
For now, the paper is best read as a controlled stress test for researchers studying evidence integration and tool-augmented systems. It is not evidence that any of the models tested is ready for high-stakes deployment.
The paper’s status
The work is an arXiv preprint, version 1, dated 20 August 2026. A journal publication is not reported. The acknowledgments report funding for DAC from an NIHR Research Professorship, a Royal Academy of Engineering Research Chair and the InnoHK Hong Kong Centre for Cerebro-cardiovascular Engineering, with support from the NIHR Oxford Biomedical Research Centre and the Pandemic Sciences Institute.
Paper data and sources
Original title: When Text and Numbers Disagree: Evidence Arbitration in Large Language Models
Authors: Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text