A tie at the top
Five leading AI models ended in a statistical tie on a newly assembled mental-health benchmark, according to an arXiv preprint. The panel means ranged from 0.610 to 0.627: kimi-k2.6 scored 0.627, followed by gpt-5.5 at 0.624, claude-opus-5 at 0.620, grok-4.5 at 0.612 and gpt-5.6-sol at 0.610. After a Holm-Bonferroni adjustment, a way to account for multiple statistical comparisons, no separation among the five survived.
The study addresses missing specialty resolution by introducing a clinician-adjudicated mental-health subset for specialty-resolved evaluation. The source was the HealthBench OSS dataset, containing 5,000 examples. The supplied document is an arXiv v1 preprint dated 25 August 2026.
How the mental-health slice was built
To find the subset, the researchers applied a screening rubric with Claude Opus 4.8 in independent batches of 50 conversations. Three licensed clinicians then independently reviewed a blinded set containing the screened-in material and 150 concealed NOT_RELEVANT controls. A conversation was included when at least two of the three reviewers agreed.
The first review round covered 791 items. It included 587 screened-in conversations and 9 of the 150 concealed controls. That produced a 6.0% control-inclusion rate, above the 5% trigger for recall, so the study ran another pass to recover candidates from the source set.
The recall loop stopped when its control-inclusion rate fell to 4.0%, below the stopping threshold. The released HealthBench-Psych subset contains 610 conversations, equal to 12.2% of the source corpus. Perinatal mental health was the largest category, with 112 conversations, or 18.4% of the subset.
What the comparison found
For the evaluation, one response per conversation was generated from each of 20 candidate models spanning eight providers. The researchers used the HealthBench messages verbatim and added no system message. Responses were stored, and every judge graded the same stored response.
The scoring system clipped candidate scores to a range from 0 to 1. Each model's panel score was the equal-weight mean of three LLM judges, and uncertainty was estimated with 1,000 conversation-level bootstrap resamples, a way of repeatedly redrawing conversations to examine how stable a result is.
On the HealthBench-Psych-Hard slice, containing 119 conversations, the same frontier cluster remained statistically indistinguishable. The figure analysis preserved the tier structure in both panels.
Agreement, refusals and a check on the setup
The judges largely agreed about which models ranked higher, even though their grading severity differed. Kendall tau, a statistic that compares the order of items in two rankings, ranged from 0.926 to 0.947 across the 20 candidates. After severity correction, estimates of judge self-preference were near zero.
The study also found a difference in refusal behavior. Only two models produced reproducible empty refusals. claude-opus-5 returned an empty refusal for 10 of 610 conversations, or 1.6%, while claude-fable-5 did so for 3, or 0.5%. Excluding those refusals yielded score estimates of 0.631 and 0.594, respectively, without changing the frontier-cluster composition.
As a check on the evaluation setup, the replicated evaluation on the complete 1,000-conversation HealthBench-Hard subset produced a score of 0.157 against the published 0.16, a difference of negative 0.003. There were no failed gradings, and all 1,000 conversations were scored.
The authors report releasing the subset, pipeline, model responses, grades and analysis code as a reusable resource. They say the study received no external funding. The release gives evaluators the materials needed to repeat the comparison on this specialty-resolved subset.
Paper data and sources
Original title: HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Authors: Matthew Flathers, Phuong Anh Nguyen, Jill Noorily et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text