A cybersecurity model named OpenAegis recorded a 58.1% Pass@1 score on CyberGym, the highest reported result in a comparison with Qwen 3.5, Kimi K2.7 and GLM 5.2. Pass@1 is the benchmark's reported rate of tasks solved in the test condition. The systems had a one-hour budget for each task. Qwen 3.5 scored 29.6%, Kimi K2.7 51.7% and GLM 5.2 43.3%, leaving reported gaps of 28.5 percentage points over Qwen, 6.4 over Kimi and 14.8 over GLM.
The study addresses a practical question: how to build a unified, open-source approach that scales cybersecurity-model training. Its CyberFactory framework links data construction, trajectory synthesis and model training across vulnerability detection, patch generation and CyberQA.
For proof-of-concept construction, instances were sourced from ARVO, OSS-Fuzz and from-the-wild CVEs. The paper does not report the total number of instances or evaluation tasks.
How the system was trained
Trajectory synthesis used a reusable, task-independent skill. In ordinary terms, the skill directed the teacher model to inspect the target and build, explore in a guided way, validate evidence and revise its approach. The skill was withheld from OpenAegis at inference, so the trained model was evaluated without it.
In CyberGym, a candidate counted as solving a task only if it triggered the target vulnerability in the pre-patch build and left the patched build unaffected. This two-sided, machine-checkable differential oracle determined whether a submission counted.
OpenAegis was initialized from Qwen 3.5-397B-A17B and fully fine-tuned for three epochs, with sequences capped at 131,072 tokens. The comparisons used the same scaffold, tools, prompt, submission logic and differential oracle, plus the same one-hour task budget; the skill was withheld at evaluation.
The model also changed its working habits
The authors also tested how the system handled long contexts. A strategy that compacted the running context at 90% produced an overall Pass@1 of 58.1%, Pass@1 on long-horizon tasks of 48.7% and context exhaustion of 7.0%. Full-history handling recorded 52.1%, 40.2% and 18.7% on those measures, while simple truncation recorded 45.6%, 36.8% and 24.5%.
A separate test looked at the skill used to generate trajectories. GLM 5.2 with the analysis skill recorded 46.5% Pass@1 across five independent 15-minute attempts, compared with 43.3% from one 60-minute attempt without the skill. The paper cautions that these conditions were not compute-matched, so the figures do not establish an equal-compute advantage for the skill.
In the with-skill condition, GLM 5.2 showed a marked change in its recorded process. Exploration coverage was 99.85%, versus 3.78% without the skill; validation coverage was 98.41%, versus 0.13%. Exploration calls averaged 2.06 per trajectory versus 0.05, and validation calls 2.17 versus 0.001. Operations per shell call were slightly lower, at 4.67 versus 5.27.
When OpenAegis was evaluated without the skill, it averaged 1.32 exploration calls and 1.05 validation calls per trajectory, compared with 0.01 and 0.00 for Qwen 3.5. The paper presents these behavioral process measures as evidence consistent with internalization, not as direct security outcomes.
The action logs showed another difference. OpenAegis used shell calls for 89.9% of tool calls, compared with 70.1% for Qwen 3.5, and read calls for 7.3% compared with 28.4%. Single-operation calls made up 13.5% of OpenAegis's tool calls and 31.4% of Qwen 3.5's, while calls containing six to 10 operations made up 30.8% and 4.3%, respectively. Calls with more than 10 operations accounted for 9.0% and 1.5%, and shell calls averaged 5.5 operations for OpenAegis versus 2.7 for Qwen 3.5.
The study also counted instrumentation events, including ASAN compilation and sanitizer-output checks. OpenAegis recorded 1,795 ASAN compilation events versus 155 for Qwen 3.5, and 2,099 sanitizer-output checks versus 1,281. It produced exactly one submission in 48.2% of trajectories, against 37.9% for Qwen 3.5, while the share with at least five submissions was 2.0% versus 10.4%. The reported behavior was more instrumented and selective.
What the benchmark does not settle
The authors describe CyberFactory as incomplete. The evaluation is bounded by the available CVE artifacts, the coverage of the benchmark and a fixed one-hour budget. Some targets still benefit from manual input construction rather than a fuzzing-first strategy, and patch generation and CyberQA have not received equally rigorous evaluation.
No statistical uncertainty estimates are reported for the benchmark differences. The result is therefore a finding within the reported CyberGym comparison and trajectory analysis, rather than a complete account of the framework's capabilities.
Paper data and sources
Original title: CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild
Authors: Jian Yang, Haau-Sing Li, Shawn Guo et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text