A test-time defense for large language models keeps a persistent external memory of rules about attack methods. Across the listed model-attack cells, its displayed attack-success rates ranged from 0.1 to 11.5 under a refusal-based measure and from 0.1 to 3.5 under the paper's primary language-model-judge measure. On benign tasks, reported scores stayed numerically close to a no-defense comparison, but steady-state use required three model calls per input.
The manuscript is a preprint identified in its front matter as arXiv:2608.26008v1, dated 26 Aug 2026.
Rules focus on attack methods
The framework operates at test time. Its induced rules describe the attack method rather than the harmful topic, with the stated aim of generalizing across attacks that share the same wrapper, or attack format.
The memory is not updated after every prompt. Rule induction and memory updating occur only after a detected violation on a non-benign input.
Under the default evaluation, memory starts empty and prompts follow a fixed order. Each input can use only rules learned from earlier failures in the same stream, making the setup a sequential interaction test.
What the benchmark measured
The evaluation uses 520 harmful-behavior prompts from AdvBench and tests benign utility on MMLU and GSM8K.
Attack-success rate is counted in two ways. The paper treats ASR-gpt, based on a language-model judge, as its primary defense metric. ASR-rej uses a refusal-phrase check; the paper cautions that this measure can underestimate success when explicit refusal phrases are injected.
Across the listed model-attack cells, the proposed method's displayed ASR-rej values range from 0.1 to 11.5, while its ASR-gpt values range from 0.1 to 3.5. The reported figures are point estimates; no confidence intervals, inferential tests or formal uncertainty measures are supplied.
For the three listed model rows, MMLU accuracy with the defense was 66.8, 61.7 and 94.2, compared with 68.7, 60.8 and 95.1 without it. GSM8K accuracy was 70.1, 83.5 and 95.0 with the defense, versus 71.8, 83.8 and 97.3 for no defense. Benchmark sample sizes and uncertainty estimates were not reported.
At steady state, each input incurs three language-model calls. The measured overhead is 3.07 times the calls and 1.43 times the latency of a single-shot model.
A sequential run reports near-zero rates
In a separate sequential analysis, attack samples arrived in batches of 20. From round 1 onward, the text reports attack-success rates near zero across all four attack types, while the no-defense baseline remained consistently vulnerable. Exact round-by-round values are not supplied, so this is a reported trajectory rather than a precise estimate of change over time.
On CodeChameleon attacks against Qwen2.5-7B and Llama3.1-8B, variants without Trigger, Enforcement or Reflection had substantially higher attack-success rates than the full model. The supplied text does not give the ablation point values.
The authors also report robustness under an adaptive composite-wrapper attack designed to evade rule triggering, along with no increase in over-refusal as memory grows. Numerical results for that attack and the XSTest over-refusal evaluation are not supplied.
The test has clear boundaries
The evaluation covers four representative black-box, prompt-level jailbreak families, plus one adaptive composite-wrapper test. Full attack coverage remains unresolved, and the results do not settle performance in multi-turn or agentic settings.
The supplied analysis identifies long interaction streams as an open issue. Explicit rule pruning and conflict resolution are not included, and the classifier and violation detector are prompted language-model components that can inherit judgment errors, including occasional over-refusal.
The supplied material describes a defensive purpose and evaluation-only use of harmful prompts, but does not report funding. It states that full prompts for all agents are provided in Appendix A.6.
Taken together, the report describes a limited computational finding: persistent, selectively updated rule memory was evaluated alongside low measured attack-success rates in the listed tests, while benign benchmark scores remained numerically close to no-defense values. The report leaves open how far those results extend beyond the tested attack families, longer streams and the excluded multi-turn and agentic settings.
Paper data and sources
Original title: A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks
Authors: Tongyan Hu, Bryan Hooi
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text