Adding more safeguards to a language model did not clearly improve protection in a new preprint, even though a seven-layer pipeline had residual attack success of 0.000 in its reported run. The same pipeline refused 81% of benign prompts, compared with 26% for Llama Guard alone, while the two configurations had nearly identical residual attack results.
The result points to a central problem in layered AI security. Defenses are often treated as if their failures were independent, so their individual attack-success rates can be multiplied to estimate the protection of the whole stack. In this study, all fifteen measurable defense pairs instead showed positively correlated failures, ranging from 0.30 to 0.75. Their joint residual attack success exceeded the multiplicative prediction by as much as 0.172.
What the test measured
The authors first mapped published LLM defenses according to how much access an adversary needs, then assessed how combinations behave using the AATM framework and a five-class model of inference-time overhead. The experiment set out three hypotheses, evaluated seven defenses against a shared adaptive adversary, included two attack classes and two static baselines, and tested the assembled stack directly.
The primary evaluation used Vicuna-7B-v1.5 with 100 harmful JailbreakBench behaviors and a matched benign set. The authors also ran a replication on Llama-2-7b-chat, while a smaller Llama-3.2-3B-Instruct run was limited to a specific comparison. The correlation findings therefore rest mainly on one target model, one behavior set and one attack seed.
The pattern held, but its cause remained unsettled
The researchers then asked whether the correlations simply reflected some jailbreaks being harder than others. After behavior-difficulty stratification and Benjamini-Hochberg correction, only one of the fifteen pairs remained significant in the reported-label analysis. The primary token-surface pair fell to a within-stratum odds ratio of 1.39, with p = 0.907.
That result weakened the case for a specific mechanism behind the correlated failures. The authors retired their mechanism-specific interpretation because the remaining association changed when they used majority-vote grader labels and external thresholds, and they said the question could not be settled from these data.
The broader composition result was more stable. Across three robustness checks, every pair remained positive. The analysis also used 10,000 marginal-preserving permutation resamples, while repeated grading based on five judgments left all fifteen pairs positive and significant. The size of the correlations changed with grading and threshold choices, but their direction did not.
A filter that stopped one attack failed against another
The defenses did not perform uniformly across attack styles. Against adaptive GCG, the perplexity filter blocked all 100 tested prompts and had residual attack success of 0. Under fluent AutoDAN-style attacks, it blocked none of the 100 prompts and had residual attack success of 0.66. The result is specific to the tested attack strategies.
Direct testing mattered more than the estimate
The study also compared predictions made by intersecting each defense's results with attacks against the assembled pipeline. For all seven defenses, the intersection estimated residual attack success at 0.010, while direct attack on the full stack found 0.000, with the two approaches agreeing on 99 of 100 behaviors. Removing Llama Guard produced a larger gap: the six-layer intersection estimated 0.190, but direct attack again found 0.000, with no feasible prompt found in the stated search across all 100 behaviors.
The direct comparison with Llama Guard led to a similarly cautious conclusion. The stack and the strongest single layer differed on only one behavior in the paired analysis, with an exact p-value of 1.0. The authors described the two results as statistically indistinguishable for the tested target and adversary.
The study's comparisons support a practical rule: isolated defense scores should not be multiplied to estimate stack protection. The assembled stack's security and refusal burden need to be measured directly.
A narrow test with a public record
The authors caution that the correlation analysis is limited. It rests mainly on the single primary target, one attack seed and an underpowered stratified analysis. The Llama-2 replication used the first 50 behaviors and was directionally consistent rather than independently confirmatory.
The document is an arXiv preprint, version 1, dated 28 August 2026, and marked as submitted to Elsevier. The authors state that the code and experimental data are available in the defense-correlation repository, including defense implementations, thresholds, attack harnesses, breach vectors, grading materials and analysis scripts; model weights are not redistributed.
The acknowledgements report support from SDAIA, KFUPM and the National Cybersecurity Authority through named artificial-intelligence and cybersecurity research grants. The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.
Paper data and sources
Original title: Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layers
Authors: Abrar Alotaibi, Muhammad Shahid Jabbar, Sadam Al-Azani, Moataz Ahmed
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text