Showing an AI evaluator a previous score can be associated with a sizable change in the rating it gives, according to a preprint that tested eight language models. Seven of the eight models had task-stratified bootstrap intervals for the anchored-metadata contrast below zero, with Llama-3.1-8B the exception. Descriptive standardized effects were -0.71 for Llama-3.2-3B and -0.67 for GPT-4.1, while Claude-4.5-Sonnet had the largest absolute score shift at -0.706 points. Because the anchored package changed more than one metadata field, the result does not show that a prior score alone caused the difference.
What the judges saw
The experiment compared three conditions: C0 with no metadata, C1 with revision framing, and C2 with revision, attempt and prior-score fields. Judges rated one candidate output on a 0-to-5 scale, with 4.0 as the acceptance threshold. Anchors were randomized independently of answer quality and presented as genuine prior evaluations; the authors restrict that claim to independence under the tested information state, rather than models knowingly following random values.
The numerical benchmark comprised 192,000 attempted evaluations and 185,271 successful ones across eight evaluated models and 20 fixed texts. The analysis retained 1,920 valid model-task-temperature-condition cells, while 59,136 complete triplets supported a descriptive standardized effect size called Cohen's d. Researchers estimated condition means from repeated decodings, averaged four temperature contrasts into 20 task effects, and ran 10,000 stratified task bootstraps, resampling five task IDs per category and treating tasks as the inferential units.
The shift reached the pass line
The same pattern appeared at the acceptance boundary. At the score threshold of 4.0 or higher, C2 was associated with lower estimated acceptance in six of the eight models: -22.29 percentage points for Claude-4.5-Sonnet, -14.40 for GPT-4.1, -8.91 for Qwen2.5-7B, -5.74 for Gemma-2-9B, -4.40 for DeepSeek-R1 and -3.06 for Llama-3.2-3B. The estimated change was +0.13 points for Llama-3.3-70B and +4.19 for Llama-3.1-8B, and uncertainty did not distinguish either increase from baseline.
Effects differed by task category. Mean total shifts were -0.047 points for code review, -0.282 for creative writing, -0.338 for factual question answering and -0.322 for summarization, making code review the weakest on average. Changing the sampling temperature did not produce a steady trend: the descriptive mean effect was -0.267 at temperature 0, -0.257 at 0.5, -0.206 at 1 and -0.258 at 2. The corresponding temperature contrast intervals included zero.
A closer look at the ratings
A selected GPT-4.1 creative-writing probe showed a score redistribution rather than a strong response to higher anchor values. Researchers ran 100 C0, 100 C1 and 100 C2 calls at each anchor value 1.0, 2.0, 3.0 and 3.9. Mean scores were 5.00, 4.80 and 4.358, respectively, so the C2-minus-C0 difference was -0.642. Under C2, probability shifted from output digit 5 toward 4, while the within-C2 slopes were approximately zero: +0.0002 for 4 and -0.0002 for 5.
The same token pattern was only partly replicated in three additional model probes. Llama-3.2-3B showed a score change of -0.120 and a -0.019 change in the probability of its most likely score token; Llama-3.1-8B changed by -0.018 with no change in that probability, while Llama-3.3-70B changed by +0.016 with no token redistribution. A ceiling analysis initially produced a coefficient of 0.211, with an interval from 0.013 to 0.408, but after adjustment the absolute-effect coefficient was -0.016, with an interval from -0.139 to 0.107. The adjusted analysis did not support ceiling proximity as a sufficient explanation.
The pattern appeared in an industry workflow
In a separate industry check, the researchers used 441 proprietary messaging-campaign samples with human labels in three classes. The original Llama-3-8B classifier had 49.66% accuracy. Group A contained 222 cases the original classifier had misclassified. Group B contained 219 cases it had classified correctly, including 119 drifting and 100 shaft samples. For Group B, GPT-4.1 made paired C0/C2 evaluations at temperature 0, with a fixed sample-specific wrong anchor.
For Group A, the correction rate was 21.62% under C0 and 22.97% under C1, but 11.26% under C2. The C2-versus-C0 difference was a decline of 10.36 percentage points, or 47.9% relative to C0. The C2-versus-C1 contrast cannot isolate the prior label, because C2 also adds an attempt field and restructures the metadata template.
Among the paired Group B cases, accuracy fell from 76.26% in C0 to 68.49% in C2, a change of -7.76 percentage points; the interval for that change was -12.33 to -3.20. Agreement with the assigned wrong anchor rose from 11.42% to 18.72%, a 7.31-point increase with an interval of 3.65 to 11.42. Of the 167 cases that were correct in C0, 17 switched to the assigned wrong label in C2: 10.18% of those cases, or 7.76% of all 219 samples.
The tested fixes were not simple
Neither of two prompt additions reduced the total difference in one numerical test. On one fixed GPT-4.1/task pair, the researchers made 100 C0 and 100 C2 calls at each of four prior scores, and all 1,500 calls were valid. The total shift was -0.4285 under the baseline prompt, -0.6327 with a chain-of-thought reasoning prompt, a 47.7% larger absolute effect, and -0.4571 with an explicit warning to disregard metadata, 6.7% larger in absolute terms. The warning flattened the within-C2 slope from 0.2674 to -0.0066, but mean total-token use rose 6.2%; chain-of-thought raised it 17.7%.
In the paired industry test, chain-of-thought was associated with a -5.48-point accuracy change, with an interval from -10.50 to -0.46, and 13.42% of 149 C0-correct cases became the wrong anchor. The warning had an estimated zero accuracy change, with an interval from -5.02 to 4.58, and seven of 163 C0-correct cases switched to the wrong anchor, or 4.29%, with an interval from 1.31 to 7.64. Relative to baseline, the warning's paired accuracy effect was estimated to be 7.76 points better, with an interval of 1.83 to 13.70.
What the study can and cannot say
The study's reach is limited. The main benchmark used one generated and screened answer for each of 20 tasks, five per category, and tested anchors only from 0 up to but not including 3.99. C2 changes the attempt field and metadata structure as well as including a prior score, so a matched template with an irrelevant number would be needed to separate those components. The token-level evidence came from selected probes, and the industry check covered one proprietary 441-sample domain; the reported effect sizes may not transfer elsewhere.
The authors interpret the results as evidence that prior-evaluation metadata can compromise an LLM judge's independence in both rubric scores and categorical decisions. They recommend uncertainty estimates that respect the task structure, warn against decomposing prompt fields without evidence, and call for mitigation tests tailored to each model, task and workflow.
The document is identified in its front matter as arXiv:2608.25869v1 [cs.CL], dated 26 August 2026, and also carries an ACM CIKM '26 proceedings citation. The research was supported in part by the Infobip Global Communication Platform project (PK.1.1.07.0001), part of the IPCEI-CIS consortium.
Paper data and sources
Original title: Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
Authors: Ante Kapetanovic, Kemal Altwlkany, Andro Mercep et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text