In controlled tests, a runtime safety layer recorded a sharply lower share of released responses judged unsafe, and the lower rates were accompanied by more refusals of safe prompts. In a matched evaluation, its two policies recorded attack success rates of 3.32% and 6.81% on HarmBench, compared with 39.20% for the disabled control. On WildJailbreak, the corresponding figures were 7.35% and 6.00%, against 41.90%. Attack success rate here means the percentage of released responses judged unsafe by an automated evaluator. The paper reports no confidence intervals for these benchmark percentages.
The results come from an arXiv preprint describing LMSM, a prototype designed to keep security decisions tied to individual requests even while a serving system continuously batches many requests together. The paper presents the system as a runtime layer that can complement model alignment, not replace it.
A policy layer inside the serving path
LMSM combines three pieces behind one serving substrate: calibrated evidence taken from inside the model, versioned policy evaluation, and a gate that holds output until it is authorized for release. The prototype supports Hugging Face Transformers and continuously batched vLLM, with SAE, transcoder and dense-probe backends.
The main evaluation used Qwen3-4B with thinking enabled in offline vLLM V1 on one NVIDIA H100 GPU, with as many as 32 active sequences. The two policies tested, called Checkpoint and Anytime, were separately fitted 15-rule policies built around the same taxonomy.
The benchmark campaign used 602 HarmBench prompts, 2,000 harmful WildJailbreak prompts and 250 safe XSTest prompts. Each benchmark-policy combination came from one fixed-seed generation run, rather than an average across repeated generations. The study therefore does not estimate how much those percentages would vary across repeated generations.
The main comparison was designed to isolate the runtime policies’ effects. Matched Disabled kept Checkpoint’s sensing and evaluation path but suppressed actions that would change the state, while Matched Empty Extension retained the eager vLLM integration but did no monitoring work.
Keeping requests separate under churn
One of the system tests focused on requests entering, leaving and moving between slots while generation was under way. Across 64 requests, the test included 32 slot reuses and 23 row movements. All 32 duplicate pairs preserved the same action, category and intervention-step outcomes, and the per-rule threshold-crossing results were preserved as well. New admissions began empty. In a separate stopping test, 24 selected requests stopped while 40 others continued.
Those findings support decision-level request isolation in the tested setup. They do not show that every underlying backend score was bit-for-bit identical after scheduler movement; the validation concerned whether the resulting decisions and intervention outcomes agreed.
Changing the evidence backend
The prototype was also tested with different ways of reading model-internal evidence. In a fixed-runtime substitution test, changing only the backend binding produced finite evidence for all 32 tested rows and completed all 64 scheduled calls or steps for both a dense probe and a transcoder. Reported throughput was similar: 2,439.96 tokens per second for the dense-probe binding and 2,487.63 for the transcoder, with interquartile ranges of 3.70 and 2.88 tokens per second over three measured repetitions after one warmup.
A separate artifact-backed test used 264 held-out HarmBench prompts across Gemma-3 with an SAE and Qwen3 with a transcoder. Without guarding, judged attack success was 46.2% for Gemma-3 and 48.1% for Qwen3. The category-matched bundles brought those figures to 5.3% and 7.2%, while the full six-rule bundles produced 4.2% and 4.9%.
Earlier intervention, higher cost
The two policies made different timing choices. Checkpoint intervened at its configured token-64 point in a diagnostic involving 1,600 intervention requests. Anytime intervened on 1,566 requests and had a median first-intervention step of 15. Among rows where an intervention occurred, its median avoided-token count was 48 tokens higher. The comparison describes when the policies acted, not how likely either policy was to intervene overall.
The serving measurements showed a larger throughput cost for Anytime. Against Matched Empty Extension, Checkpoint’s throughput retention was 99.31% at width 1 and 98.14% at width 32. Anytime’s was 93.35% and 83.91%; the reported width-32 overheads were 1.86% for Checkpoint and 16.09% for Anytime.
At width 32, median throughput retention was 97.88% with one active Checkpoint rule, 98.31% with six and 96.89% with 15. The observed medians did not establish a general per-rule scaling law.
What the tests can—and cannot—say
The lower harmful-output rates were accompanied by higher false-refusal rates on the safe XSTest prompts. False-refusal rate means the share of safe prompts labeled as refusals; it was 2.40% for Matched Disabled, 4.40% for Checkpoint and 5.60% for Anytime. The reported unweighted safety average—the mean of the two harmful-prompt results and XSTest—was 27.83% for the disabled control, 5.02% for Checkpoint and 6.14% for Anytime. The paper reports no confidence intervals for these figures.
The study’s stated scope is mediation correctness and measured benchmark behavior on a particular serving path, under stated trust assumptions. It does not establish that the policies themselves are correct, replace alignment or cover arbitrary token streaming. The threat model trusts the serving process, model, backend artifacts and parameters, policy configuration and enforcement; compromise of those components is outside the stated scope.
Moving to other models, activation sites or runtimes would require suitable bindings and calibration while preserving request-identity, decision and output-release invariants. The current release supports one active backend binding per policy and only allow, terminate and refuse actions; arbitrary token streaming and additional actions such as redaction or controlled regeneration were not evaluated.
External training-time comparisons were contextual rather than matched reruns in the LMSM serving path. Open questions include how the installed backends and calibration rules perform under adaptive inputs and distribution shift, whether the system’s invariants generalize across deployment conditions, and what semantics and costs would come with multiple backend bindings or new enforcement actions.
A reproducible systems claim
The authors report that the implementation and reproduction artifact is available at https://github.com/xiuyuz/LMSM. It includes the LMSM library, tests, runtime integrations and deployment profiles. The supplied document is an arXiv version 1 preprint dated 26 August 2026.
Paper data and sources
Original title: LMSM: LLM Security Framework Inspired by Linux Security Modules
Authors: XiuYu Zhang, Bonan Ruan, Junfeng Fang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text