Claude Opus 4.8 led a benchmark that asked models to generate inputs intended to trigger as many distinct crashes as possible through an instrumented harness in a self-contained Docker image. It scored 196 of 579 points, or 33.85%, and triggered a crash in 60 of 77 challenges. Sonnet 4.6 scored 156 points, or 26.94%, with crashes in 50 challenges, while Haiku 4.5 scored 58 points, or 10.02%, with crashes in 35 challenges.
None of the three models triggered a crash in 13 challenges. The comparison is reported in an arXiv version 1 preprint dated 25 Aug 2026, covering 77 challenges from 43 open-source projects.
How the test was run
Each model inspected the vulnerable-version source and a sanitizer-instrumented harness, then generated inputs intended to trigger distinct crashes inside a self-contained Docker image. Models received no bug description, patch or fix commit, and grading was deterministic and offline, with no answer key supplied.
Challenges were built from published bug reports using the vulnerable source revision, harness, sanitizer configuration and proof of concept. The images retained the vulnerable revision and excluded patches. Each model was evaluated on all 77 challenges, with a 100-turn budget and a 1,800-second limit for each episode.
A crash signature, the recorded label used for a failure, was scorable only when three executions crashed and produced the same signature. A crash was valid unless the crash site was in the harness or both standard outputs were empty.
How the score was built
The corpus covered 14 bug classes. Of the 77 challenges, 45 were memory-safety challenges and 32 were denial-of-service or other-fault challenges. By language, it included 36 C challenges, 32 C++ challenges and nine Java/JVM challenges.
The benchmark converted the results into a difficulty-weighted score out of 579. No more than three distinct crash signatures from a single challenge could contribute to a model's score. Difficulty was defined relative to a fixed panel of the three models, rather than as an intrinsic property of the vulnerability, with coefficients from 1 through 5.
The tier results were not uniform. Sonnet performed slightly better than Opus in lower-difficulty tiers, while Opus scored better in tiers D3 and D4. The overall scores placed Opus first at 196 points, followed by Sonnet at 156 and Haiku at 58.
In a separate no-cap calculation, the reported score increases were 33.16% for Opus 4.8, 30.77% for Sonnet 4.6 and 12.07% for Haiku 4.5.
What the result cannot tell us
The comparison is limited to three Claude models on 77 challenges from 43 projects. The findings support a comparison within this benchmark rather than a general verdict on AI performance in software security.
It does not show that Opus 4.8 found more distinct underlying vulnerabilities, because crash-signature counts are not equivalent to root-cause counts. A repeatable crash signature does not by itself identify the underlying cause or show that the failure is exploitable.
None of the three models triggered a crash in 13 challenges under the same source, harness and evaluation budget used for the comparison. The result identifies a part of this corpus that the tested systems did not reach, but it does not say whether a different model, a different setup or a later run would do better.
The corpus and harnesses are publicly available at [the project repository](https://github.com/fuzzingbrain/FuzzingBrain-Bench). The report leaves open how the comparison would look on a larger, more diverse corpus and how stable model rankings would be across repeated independent episodes.
Paper data and sources
Original title: FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs
Authors: Ze Sheng, Aleksandar Kezic, Zhicheng Chen, Jeff Huang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text