A preprint reports that EAVA, an AI system for assessing software-vulnerability reports, achieved the strongest overall results in a benchmark against six machine-learning, two deep-learning and three large-language-model baselines. It was ahead of proEVA on every individual CVSS metric except Attack Complexity.
Relative to proEVA, the reported gains were 5.3% in average weighted F1, 18.7% in average MCC, 14.4% in severity weighted F1 and 35.2% in severity MCC. The largest individual F1 gains were reported for Confidentiality at 9.7%, Attack Vector at 8.3%, User Interaction at 7.9% and Integrity at 6.5%.
For one individual metric, Scope F1 was 0.976 for EAVA and 0.949 for proEVA. The evaluation scored eight CVSS v3 base metrics using weighted F1 and Matthews Correlation Coefficient (MCC), then reported average scores and severity measures calculated from predicted CVSS values.
A system designed to show its work
EAVA uses three specialized LLM agents to analyze code snippets, screenshots and information about the vulnerable project. Its assessment LLM is trained in two stages - supervised instruction tuning followed by reinforcement learning - and the system retrieves similar historical vulnerabilities as supplementary evidence.
The benchmark contained 6,446 CVE-linked GitHub issue reports from 1,986 projects, spanning 54 programming languages and 159 CWE categories. NVD CVSS v3 labels supplied the reference values, with OSV references used alongside them.
The reports were split chronologically into training, validation and test sets in an 8:1:1 ratio. The benchmark compared EAVA with the baseline groups on the eight metrics and on overall average and severity measures.
To check the generated reasoning, the authors annotated 51,568 metric-level data points and had two domain experts validate a random sample of 400. Of those, 391, or 97.8%, met both factuality and relevance criteria; Cohen's kappa for agreement was 0.84, after 56 person-hours of validation.
Component tests favored the full configuration
Comparisons with stripped-down versions showed higher reported scores for the full configuration. Against the version without fine-tuning, average weighted F1 and MCC were 10.1% and 49.6% higher, while severity weighted F1 and MCC were 15.1% and 39.6% higher.
Against a version without explicit reasoning, the average weighted F1 and MCC differences were 4.2% and 20.1%. Against a version without reinforcement learning, the differences were 3.1% and 13.5%, and EAVA scored higher on every individual CVSS metric.
In comparisons that removed information sources, code processing had the largest reported severity gains: 8.5% on weighted F1 and 13.4% on MCC. The corresponding gains were 2.2% and 4.3% for screenshot information, and 4.4% and 8.9% for project information; the full enrichment configuration had the best overall performance.
Practitioners generally rated the evidence useful
The separate practitioner exercise involved eight security practitioners, 40 randomly selected reports from the test set and 320 metric-level tasks. Among 284 cases with correct EAVA predictions, participants rated the evidence useful at 4 out of 5 in 275 cases, or 96.8%.
Retrieved historical-vulnerability information was rated useful at the same level in 78 of 89 examined cases, or 87.6%.
When EAVA was wrong, the results were less clear. In 36 incorrect-prediction cases, participants selected the correct value in 12; among the remaining 24, 18 were judged to involve insufficient public information or a potentially inappropriate NVD assignment. Evidence also had lower relevance and completeness in these cases.
A promising benchmark with a narrow reach
The result was broadly comparable when the underlying 8B model changed. With Qwen3-8B, average/severity F1 was 0.872/0.665 and average/severity MCC was 0.622/0.490; with Llama-3.1-8B, the figures were 0.874/0.672 and 0.646/0.490.
But the benchmark is limited to GitHub reports linked to NVD records and to the CVSS standard. The authors flag possible inaccuracies or differences in NVD labels, as well as possible overlap between test disclosures and model pretraining data.
The practitioner study had eight participants and descriptive ratings, not a randomized evidence-versus-no-evidence comparison. The findings therefore do not establish a benefit for analyst time, remediation decisions, exploit prediction or operational security outcomes, nor do they show that the benchmark result would generalize to other vulnerability databases, report sources or assessment standards.
Paper data and sources
Original title: Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence
Authors: Shengyi Pan, Zelong Zheng, Jiayuan Zhou et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text