A preprint testing 13 frontier AI models on complete, multi-step computational-biology workflows found that even the best-performing system averaged a task score below one-half. GPT 5.6 Sol led with a mean score of 0.48. In this benchmark, an artifact was a structured output required by the task, and the task score was the proportion of those outputs that cleared the pass threshold.
The result measures execution within a defined research workflow, not whether an agent could decide which biological question to ask or which analysis to choose. The benchmark supplied those choices in advance, so success measured how well a model carried out a specified pipeline.
A test built around published studies
BixBench3 was built from 20 published computational-biology studies spanning 17 assay types and nine scientific domains, with 138 graded artifacts in all. Its central task was to reconstruct published analyses from raw biological data and produce structured artifacts corresponding to published results.
Across the evaluation, 13 models worked through 20 benchmark tasks, producing 260 completed runs and 1,794 graded artifact evaluations. Generated artifacts were compared with published reference outputs.
The leading models were close
GPT 5.6 Sol had the highest average task score at 0.48, followed by Kimi K3 at 0.47. GLM 5.2 and Claude Opus 4.8 both averaged 0.46.
The averages were built from a binary artifact pass rule: a score of at least 0.8 counted as a pass, and each task score represented the proportion of artifacts that passed. The threshold was informed by expert review of 25 artifacts.
Long analysis chains exposed a weak spot
The deepest outputs scored lowest. Mean binary artifact pass scores were 0.30 for depth-1 artifacts, 0.44 for depth-2 artifacts and 0.24 for artifacts at depth three or greater.
Raw-data size showed another sharp divide. Mean task scores were 0.37 for tasks using less than 50 GB, 0.34 for 50 to 100 GB and 0.10 for tasks using more than 100 GB.
Scores were highest in microbiology, evolution and ecology, and immunology. Neuroscience and cell biology tasks scored lower. These are task-stratified associations within the benchmark, so they do not by themselves show that analysis depth or data size caused the difference.
A single attempt took hours and millions of tokens
An individual task attempt was substantial in computational terms. On average, it used 102 million tokens, took 6.8 hours and involved 695 model turns.
Reported cost varied 367-fold across models, from $0.35 to $129.14 per task. The estimates excluded cache-storage costs. Cost and performance were broadly associated, but the highest performance was not achieved at the highest cost, with several models forming distinct cost-performance tradeoffs.
Failures tracked closely with lower scores
The benchmark's failure annotations followed the same pattern. Total failure-mode tags were strongly negatively correlated with mean task score, with Spearman rho = -0.92 and p = 9.9 x 10^-6. The highest-scoring models had 31 to 51 tags, compared with 102 to 117 for the two lowest-scoring models.
Premature termination and retry loops were about 2.0 and 2.2 times more frequent in the lowest task-score quantile. The labels were assigned by an LLM judge using concrete execution evidence, so they are automated annotations rather than independent mechanistic diagnoses.
What the scores cannot answer
The benchmark's design sets a firm boundary around the finding. Each task prescribed the objective, methods, contrasts, filters and required outputs. A model could therefore score well by executing the given plan, while the benchmark still says little about its ability to select a worthwhile research question or analysis.
A scientifically valid alternative method may receive a low artifact score if it produces a different output from the published reference. The results are therefore a test of reproduction under fixed rules, rather than a complete measure of scientific judgment.
The benchmark is available for reuse
The benchmark code is available on GitHub and its dataset on Hugging Face. Future model updates are planned on Edison Advances.
No funding source is reported in the supplied text. It also reports that all authors were employed by Edison Scientific for the duration of the study.
Paper data and sources
Original title: BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
Authors: Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text