The gap between reaching and triggering
A reported set of 24-hour fuzzing campaigns on the updated Magma benchmark reached 77 of 127 bugs, but triggered only 43, according to an arXiv preprint. In this framework, reaching means the benchmark's designated bug-checking oracle was called; triggering means the bug was reached and its trigger condition was satisfied. The report evaluates fuzzers, the testing tools used in these campaigns, by keeping those stages separate.
That distinction is central to the paper's question about how fuzzer evaluation should measure real bug finding. It argues that higher code coverage is not necessarily better bug-finding effectiveness, and asks for measures tied to bugs whose presence is already known. Magma defines three bug-centric metrics: reaching, triggering and detecting. Detection is assessed separately through replay and runtime checks, so a bug that is reached or triggered is not automatically counted as detected.
Built around real bugs
Magma begins with a set of open-source programs and bug cases. The initial artifact covered seven open-source library or application targets, about 2 million lines of code, and 118 bugs that were analyzed and reinserted. The bug set spans 11 CWE categories, was sourced from bug reports, and was forward-ported, or carried into the most recent version of each target codebase.
The benchmark also includes 25 executable drivers sourced from the original target codebases. It uses principal component analysis, or PCA, as a way to compare how varied the targets are. The reported comparison places LAVA-M targets in a cluster, while CGC and Magma show broader variety. That is a description of benchmark composition, not a measure of which fuzzer performs best.
Why v1.3 was needed
Magma's version history shows a benchmark that has been repeatedly revised. Version 1.0 in 2020 had seven targets and 118 bugs. Version 1.2 in 2021 had nine targets and 138 bugs. Version 1.3, updated in 2025, had nine updated targets and 127 bugs.
The case for the latest update was partly a maintenance problem. Of the 138 older bugs, 57 could no longer be applied, while 246 new CVEs affecting Magma's targets had been found since v1.2. The 2025 update moved targets and fuzzers to recent versions, retired 11 broken patches, improved debugging and validation infrastructure, and added automated versioning and patching plus a proof-of-concept mode.
What the campaigns measured
To test the update, the reported campaigns used 24-hour runs with AFL++, Honggfuzz and libFuzzer on nine targets. Runtime monitoring recorded bugs that were reached or triggered. Crash inputs were then replayed against benchmark canaries and ASAN-instrumented targets, providing a separate check on detection.
In separate tool-specific counts, AFL++ triggered 40 bugs, compared with 28 for Honggfuzz and 10 for libFuzzer, giving AFL++ the highest observed count. The paper presents this as a descriptive comparison, not evidence of causal superiority. Because the campaigns lasted 24 hours and covered nine targets with three named fuzzers, the result belongs to that reported setup.
The benchmark-wide summary lists 77 bugs reached and 43 triggered out of 127. Among the 57 bugs ported from older versions, 91% were reached and 34% were successfully triggered. The reported update results define detection as a separate outcome, but do not provide a comparative detected-bug count for these campaigns.
A benchmark with a growing footprint
The benchmark's reported community-use indicators extend beyond the campaigns. Through September 2025, it had 286 citations, 108 forks, 319 stars and 78 pull requests. The paper also reports at least 45 high-quality papers using Magma since its 2021 release, including 14 in 2024 and 14 in 2025. Those figures show community activity and research uptake, but they are adoption indicators rather than direct measures of benchmark effectiveness.
For readers trying to compare fuzzers, the main value is the separation of questions. Did a campaign reach a bug? Did it satisfy the condition that triggers it? Was the fault detected? Magma v1.3 makes those distinctions visible, but the reported campaign does not establish that one tool causes better results, that the selected targets represent all software, or that higher coverage is a reliable sign of better bug-finding effectiveness. Its fuzzer comparison is best read as an observation from the reported 24-hour campaign on nine targets.
Paper data and sources
Original title: The Impact of Magma: A Ground-Truth Fuzzing Benchmark
Authors: Ahmad Hazimeh, Adrian Herrera, Srividya Subramanian et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text