In clean RedCode benchmark tests, a fixed SkillShield security skill in the coding agent's system prompt was associated with lower observed harmful tool-execution and malware-generation scores than no defense. The proactive all-classes setting had a 43.6% mean execution attack-success rate and a 0.58 mean generation score, compared with 66.2% and 3.28 without a defense.
Llama Guard 3, a separate comparison condition, recorded a 42.7% mean execution attack-success rate and a 1.40 mean generation score. The averages were calculated across six LLMs, so they describe the tested group rather than every coding agent.
A defense written into the prompt
SkillShield is a prompt-space defense: its complete security skill is placed in the system prompt before session start, giving the policy coverage from the first step while using prompt space on every request.
Skill generation used a 10,000-character body budget, or about 2,500 tokens, and refinement in 2,000-character chunks. The study tested all-classes, per-bundle and per-class provisioning, with those scopes fixed before requests rather than selected by adaptive runtime routing.
Agent runs were capped at 10 steps, a $0.10 cost limit, up to 1,024 output tokens per call and a 600-second timeout in isolated Docker containers.
The tests measured execution and generation
The malicious execution benchmark, RedCode-Exec, contained 27 task groups with 30 base cases each, for 810 base cases. RedCode-Gen contained 160 cases across eight malware families.
Execution scoring assigned 0 for refusal, 1 for an unsuccessful attempt and 3 for a successful attack. Generation was scored by a single judge using a 0/1/5/8/10 rubric, producing the average generation score reported in the comparisons.
Observed rates varied by policy scope
Across the fixed scope comparisons on clean RedCode, the reported mean execution rates were 40.1% for reactive all-classes, 35.4% for reactive per-bundle and 10.1% for reactive per-class. At per-class scope, proactive and reactive settings both had a mean generation score of 0.42.
The scope-transfer results show why a single headline number needs context. Per-class provisioning had a 6.0% in-scope attack-success rate, compared with 34.1% for all-classes. Per-bundle provisioning recorded 37.0% in-scope and 44.8% outside-bundle, while all-classes recorded 43.9% on an unseen class, compared with 79.0% for the undefended agent.
Those comparisons used fixed deployment assumptions: the policy scope was known before requests, and the study did not evaluate adaptive runtime selection. Per-bundle provisioning was measured only for execution, with no generation result reported for that scope.
Other comparisons were less decisive
The prompt budget was also compared in two arrangements. Budget-matched concatenation had a 26.5% mean execution attack-success rate, versus 36.2% for the synthesized bundle, and was better on five of the six LLMs. The reported six-model comparison was not statistically significant.
At per-class scope, the reactive setting had lower observed execution rates than the proactive setting on every model, with differences ranging from 1.6 to 6.9 percentage points. The means were 10.1% for reactive synthesis and 14.5% for proactive synthesis, while the difference was less stable at broader scopes.
Fixed jailbreaks did not erase the gap
Under two fixed reformulations of malicious requests, the reactive per-class condition continued to show lower observed scores than no defense. Under AgentHarm, execution attack-success rate was 31.2% and generation score was 0.65, compared with 68.3% and 1.65 without a defense. Under ArtPrompt, the corresponding figures were 13.8% and 0.61, compared with 47.4% and 1.53.
The tests used fixed, unseen reformulations rather than attackers that adapted their prompts while targeting the deployed skill. Statistical support was stronger for execution than for generation: comparisons with p below 0.01 fell from 54 on clean tasks to 48 under AgentHarm and 49 under ArtPrompt for execution; generation counts fell from 31 to 19 and 12.
The paper links the weaker generation pattern to the smaller eight-family benchmark, tied pairs and the resulting smaller number of untied comparisons.
The benign check had a narrow meaning
On 731 benign SWE-Bench Pro tasks, every baseline had zero safety refusals across six LLMs, while SkillShield had zero on four LLMs. No informative paired p-value was reported.
This was a narrow check of safety-grounded refusal, not a test of whether agents completed benign software tasks end to end. The result therefore offers limited evidence about utility beyond the specific refusal behavior measured.
What the benchmark leaves open
On OpenCode and Pi, the observed figures with reactive skills were lower than without the skill among the four LLMs included in the cross-harness comparison. OpenCode execution attack-success rate was 20.3% versus 71.3% without the skill, and generation score was 0.78 versus 3.72; on Pi, the corresponding figures were 18.4% versus 65.2% and 1.51 versus 4.25.
Taken together, these are controlled benchmark and harness results, not evidence from live deployment. The jailbreak evaluation used fixed, non-adaptive reformulations, and the prompt-space policy depends on the model continuing to follow the system-prompt instructions; it is not an independent action-enforcement gate.
Matched results were compared with two-sided paired Wilcoxon signed-rank tests. The analysis used a Bonferroni-adjusted threshold of 0.01, classified an absolute rank-biserial correlation of at least 0.5 as a large effect, and calculated percentile 95% bootstrap confidence intervals from 10,000 resamples.
The paper states that supporting code and artifacts will be released publicly upon publication.
Paper data and sources
Original title: SkillShield: Prompt-Space Security Skills for LLM Coding Agents
Authors: Xiaodong Wu, Zhimin Zhao, Qi Li et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text