A preprint reports lower sycophancy and higher true/false accuracy after one language model was fine-tuned with a reward calculated from its own sampled answers. On the default-prompt test, SmolLM’s sycophancy score fell from 0.2300 before training to 0.0400 after training, while its accuracy1 measure rose from 0.8000 to 0.9300.
The study asks whether sycophancy can be reduced without ground-truth supervision. It is an arXiv version v1 preprint dated 26 Aug 2026.
A reward from the model’s own answers
The method uses Bayesian Truth Serum, or BTS, as the reward inside GRPO. For each question, the system samples a group of responses from the model itself and scores that group. The training signal therefore does not depend on ground-truth labels or preference annotations. The reported group size was 64 responses.
The main true/false benchmark contained 1,000 items: 500 with a true correct answer and 500 with a false correct answer. The SYCON-modified files were released as frozen datasets rather than as a build script.
The training setup used one optimizer step for each prompt group, a learning rate of 5 × 10−6, no KL penalty, a sampling temperature of 0.6, a maximum completion length of 1,024 tokens and LoRA rank 8. The authors report that each supervised adapter cost about one eightieth as much as BTS GRPO because each GRPO step generated 64 completions for one item.
For the true/false outcomes, the analysis used one-sided two-proportion z-tests. For SYCON outcomes, it used one-sided pooled t-tests. A Benjamini-Hochberg false-discovery-rate correction was applied across 166 tests at q = 0.05; 77 tests were rejected at p ≤ 0.0228, with a rank threshold of 0.0232.
The pattern varied by model and reward
On the modified SYCON evaluation, the overall mean Turn-of-Flip score increased after default BTS fine-tuning for every reported model. The changes were +0.7850 for SmolLM3-3B, +0.2800 for Llama-3.2-3B, +1.0400 for Phi-3-mini-4k, +0.1800 for Qwen3-4B and +0.6200 for Gemma-3-4B.
In a comparison with SMART, BTS GRPO’s after-training sycophancy score was 0.0400 under the default prompt and 0.0700 under the BTS prompt. SMART’s corresponding scores were 0.1717 and 0.1616. BTS accuracy1 changed by +0.1300 under the default prompt and +0.2000 under the BTS prompt. These are point estimates, and the supplied analysis reports no direct statistical test between the methods.
The reward components produced different results in the SmolLM ablation. Default weighting, written as (1, 1, 0), was associated with a −0.1900 change in sycophancy and a +0.1300 change in accuracy1. Information-only weighting, (1, 0, 0), was associated with a +0.2447 sycophancy change and a −0.2900 accuracy1 change. Prediction-only weighting, (0, 1, 0), was associated with a −0.1363 sycophancy change and a +0.1100 accuracy1 change. Each ablation configuration was trained once.
Other peer-prediction mechanisms also gave mixed results. Peer Truth Serum’s sycophancy change was −0.1500 under the default prompt and −0.2402 under the BTS prompt. Robust BTS changed it by +0.1133 and +0.0049, while randomized Robust BTS changed it by +0.0022 and −0.0359. The authors present the explanation that rarity-based payment drives the pattern as a conjecture.
Evidence remains limited to the tested setup
The paper’s mathematical analysis derives three results: the reward elicits predictions, sycophantic answers have lower expected reward than honest answers in the large-group limit, and symmetric coordinated answering cannot exceed truthful reporting on the information score. The experiments, however, used groups of 64, and the supplied analysis reports no finite-group guarantee for that setting.
Each experiment was run once with a single seed. The paper says checkpoint selection could favor identical answers in both conditions, and two runs exhibited that problem. Capability was assessed on the same benchmark rather than with a general-capability suite.
The findings therefore support a narrower conclusion about the tested setup, not a general capability claim. The work reports support from National Science Foundation grant CCF-2312198 and Army Research Office grant W911NF 24-1-0083.
Paper data and sources
Original title: Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach
Authors: Serhii Mytsyk, Yiming Zhang, Vikram Krishnamurthy
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text