Fluent AI-generated vulnerability reports can look more convincing than the evidence they contain, according to an arXiv preprint reviewing research on language-model failures in security work. The paper frames the problem as a mismatch between human causal, multi-step reasoning and the probabilistic token generation used by autoregressive language models, rather than simply as a question of who wrote the report.
The review searched IEEE Xplore, the ACM Digital Library and arXiv, then deduplicated results and had two independent reviewers screen them. It retained studies that supplied empirical evidence of language-model failure modes in security contexts and passed full-text relevance and experimental-rigor filtering. The review did not report how many records were retrieved, screened or retained.
A problem with several faces
Its taxonomy divides AI slop into three broad forms: hallucinated vulnerabilities, incorrect patches and semantic repackaging. In the first, a model describes a weakness that is not actually present; in the second, it proposes a faulty fix; in the third, it repeats familiar security language without a grounded account of what happens in the program.
The review points to a HackerOne example to show why the issue can matter operationally. It reports that by mid-2025 roughly 20% of the reports in that case were identified as low-quality AI slop, while valid reports were around 5%. In January 2026, Stenberg announced that the program would be discontinued. The paper presents this as a cited case, not as a measure of AI slop across vulnerability-triage systems; its underlying denominator and independent verification procedure were not reported.
The missing link is execution
At the center of the paper's argument is an execution gap. The review says none of the cited studies provided execution-level validation of claims in language-model outputs. Compile checking and static analysis were the strongest forms of grounding it found.
The authors link that gap to a difference in how the systems produce answers. Human security reasoning is described as causal and multi-step: a reviewer must connect a particular program behavior to a condition that can actually be exploited. An autoregressive language model instead generates probable next tokens. The paper treats that mismatch as a structural reason a polished explanation can still fail to describe a real exploit.
The review says chain-of-thought prompting and tool-using agents can narrow the reasoning gap, but not close it. Retrieval-augmented generation, or RAG, can anchor a model to factual material, the paper says, yet it cannot establish that the specific code being examined instantiates the retrieved vulnerability class or prevent logical hallucination.
Verification before judgment
The proposed answer is to make generated findings pass active verification before they reach a human triager. The pipeline is designed so that a hallucinated claim must survive at least two independent verification stages; the paper says most slop fails at the first hurdle. The architecture is proposed, however, and no end-to-end performance result is reported.
That approach puts correctness ahead of provenance. Passive detection and watermarking target where a report came from rather than whether its exploit path is true, and the paper argues that they face fundamental entropy constraints. Cryptographic attestation can establish who submitted a report, it adds, but not whether the exploit path is real.
The paper's preferred workflow treats generative output as a hypothesis and applies deterministic checks before human judgment. That recommendation is aimed at trustworthy triage, but it does not show that the proposed stack is complete or reliable for complex programs. The authors caution that every component has blind spots, including incomplete static-analysis coverage, constraints on fuzzing paths and scalability limits for symbolic execution, with no guarantee that repeated checking will converge on self-correction.
Two yardsticks, still on paper
To make the fluency-substance gap measurable, the authors propose the Deductive Coverage Score. It is intended as a proxy for how explicitly and verifiably a generated vulnerability claim is grounded. No empirical calibration, threshold or validation result is reported.
They also specify CVE-Bench, which would compare verified CVE reports with fluent language-model reports on the same codebases that cite nonexistent CVEs or fabricate exploit paths. The proposed evaluation includes measures for classifying phantom reports, locating ungrounded claims and ranking triage priorities. The review does not report benchmark results.
A second proposal, Slop-Score, combines normalized fluency, evidence density and constraint-satisfaction ratio. Its purpose is to quantify how polished a report is relative to the evidence and constraints it meets. Like the benchmark, the score remains a specification: no validation dataset or discrimination result is reported.
A map, not a safety guarantee
The review's reach is narrower than its subject. It is a systematic survey and conceptual roadmap based on cited literature and case material, not a new trial, pooled quantitative synthesis, benchmark run or end-to-end evaluation of the proposed pipeline. It reports no pooled effect estimates, confidence intervals or inferential meta-analysis.
That means the paper cannot say how common AI slop is across vulnerability-triage systems or whether the cURL example generalizes. It also does not establish that language models caused particular triage failures, that any proposed mitigation reduces them, or that the new scores have a useful decision threshold. Those questions remain open: the proposed instruments need expert or deterministic ground truth, and active verification needs testing in realistic security pipelines, including against attempts to exploit the blind spots of its components.
For now, the document is an arXiv preprint, version v1, dated 26 August 2026. Its value is as a map for building and testing more trustworthy AI-assisted security triage, not as evidence that the proposed safeguards already work.
Paper data and sources
Original title: AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation
Authors: Junchen Ding, Jialiang Dong, Yichen Zhu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text