A benchmark of malicious AI agent Skills found that text-based detectors that scored well on random tests performed substantially worse when tested on Skills from sources held out of training.
On the source-held-out test, the strongest word TF-IDF SVM reached a Macro-F1 of 0.665—a summary score that gives equal weight to the malicious and benign labels—while catching 95.6% of malicious Skills. It also falsely flagged 62.4% of benign Skills, the model’s benign false-positive rate: the share of benign items it marked as malicious.
Built to expose duplication and source effects
The authors froze 13 public sources and retained 11 Core-contributing sources. After deduplication, 8,414 raw malicious artifacts became 7,539 normalized-unique identities organized into 4,588 operational structural families.
Cross-label conflict exclusion removed 34 malicious normalized units and 3 Main-benign units. The final benchmark contained 9,740 units: 7,505 malicious and 2,235 Main-benign.
The study compared several ways of splitting those units. In the Random protocol, 70% went to training, 10% to validation and 20% to testing; the malicious structural-family-disjoint protocol kept each of the 4,588 families together. In the Source-Disjoint protocol—meaning the test sources were held out of training—the held-out sources were SRC009, SRC011 and SRC012.
Learned baselines read only inert primary Skill instruction text, using word- or character-level TF-IDF features with linear classifiers. The study also tested three public scanners, including SkillSpector in a mode with its LLM disabled.
The gap appeared beyond random splits
On Random splits, learned baselines reached Macro-F1 scores of 0.882 to 0.932. Keeping malicious structural families separate reduced Random performance by 0.016 to 0.038, while Source-Disjoint scores fell to 0.653 to 0.665.
That drop was visible in the error counts. The word-SVM flagged 340 of 545 benign test Skills and missed 37 of 839 malicious test Skills; SRC011 contributed 293 false positives.
The paper cautions that Source-Disjoint evaluation does not isolate a causal estimate of universal unseen-source difficulty. Holding out a source also changes provenance, construction, documentation and label mixture, so the result cannot be read as a general causal effect of source identity alone.
Different tools made different trade-offs
Fixed scanner results showed the same tension from another angle. Cisco detected 2.5% of malicious Skills and falsely flagged 1.1% of benign Skills; SkillSpector detected 0% of malicious Skills with a 0.55% benign false-positive rate. SkillFortify detected 25.3% of malicious Skills, but its benign false-positive rate was 49.9%.
Class balancing improved the word-SVM’s Source-Disjoint Macro-F1 from 0.665 to 0.710 and cut its benign false-positive rate from 62.4% to 43.3%. Scaffold sanitization did not close the gap: Macro-F1 stayed between 0.643 and 0.662, while benign false-positive rates ranged from 62.4% to 64.0%.
A static benchmark with bounded coverage
Attack labels covered only part of the malicious set. Supported attack mappings covered 4,983 of 7,505 malicious identities, and the separate derived-impact mapping covered 2,128 identities. Among mapped identities, Execution / Code Delivery was most common at 3,320, followed by Instruction / Goal / Memory Manipulation at 1,671 and Privilege / Tool / Authority Abuse at 1,013.
All 9,740 benchmark identities have public readable representations, but exact frozen static text is available for 9,735; five malicious records have sanitized representations.
The benchmark is static in these experiments: its contents were treated as inert data, and the study did not execute code, helpers, payloads, URLs, installers or embedded commands. The results therefore describe the tested static artifacts and configurations, not runtime behavior.
The manuscript is a version 1 arXiv preprint dated 20 August 2026.
Paper data and sources
Original title: MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection
Authors: Yue Wang, Yi Liu, Gelei Deng et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text