In a controlled test of Linux authentication behaviour, Meta Llama 3.1 8B Instruct and Qwen 2.5 7B Instruct detected a larger share of anomalous scenarios than Wazuh and OpenSearch. Meta Llama delivered the strongest observed overall results, while GPT-OSS often failed to return a valid structured response.
A controlled test of messy signals
The comparison ran inside an isolated virtual network with a Wazuh monitoring server, a monitored Linux endpoint and a Kali Linux attacker machine.
The curated dataset contained 75 representative behavioural scenarios: 20 normal, 24 borderline or suspicious, and 31 anomalous. Suspicious and anomalous cases were intentionally overrepresented to test sensitivity and the rate at which systems missed anomalies.
To create the main binary scorecard, the study treated severity scores of 0 or 1 as Normal and scores from 2 through 5 as Anomaly. That left 24 Normal and 51 Anomaly scenarios, and every detector was compared with the same ground-truth classifications.
Each language model received the same behavioural description, classification requirements, severity definitions, seven behavioural cues and JSON response structure. The models were run with temperature set to 0 and a maximum response length of 300 tokens.
The models split on missed cases
On the full 75-scenario scorecard, Meta Llama recorded 89.3% accuracy and 88.2% recall, meaning it detected that share of the benchmark's anomaly cases. Its false-negative rate was 11.8%, and it had the strongest observed combination of accuracy and recall in the comparison.
Qwen recorded 77.3% accuracy and 68.6% recall. It detected 35 anomalous scenarios and missed 16, with a false-positive rate of 4.2% and a false-negative rate of 31.4%. Its lower false-positive rate than Meta Llama's 8.3% was paired with weaker anomaly detection.
Wazuh detected 16 of the 51 scenarios classified as anomalies, while OpenSearch detected 13. Their recall was 31.4% and 25.5%, respectively; the systems missed 35 and 38 anomaly scenarios, with false-negative rates of 68.6% and 74.5%.
Borderline cases exposed the gap
The difference was also visible in the borderline cases. Among 20 severity-2 borderline anomalies, Meta Llama detected 16 (80.0%), Qwen 11 (55.0%), Wazuh four (20.0%) and OpenSearch three (15.0%). GPT-OSS detected five cases (25.0%) end-to-end, while 15 cases produced no valid structured output.
Accuracy was not the whole story
Speed and output format changed the practical picture. Qwen averaged 0.83 seconds and Meta Llama 1.86 seconds, and both returned valid structured responses for all 75 scenarios. GPT-OSS averaged 4.94 seconds and returned 22 valid responses and 53 invalid ones, for a validity rate of 29.3%.
Among its 22 valid responses, GPT-OSS got 21 correct, producing 95.5% accuracy and 94.1% recall. Those conditional figures are not directly comparable with the complete 75-scenario results, because invalid structured outputs counted as unsuccessful end-to-end classifications rather than Normal predictions.
That output problem persisted under unchanged conditions. Across three runs, GPT-OSS produced 22, 29 and 29 valid responses, equal to validity rates of 29.3%, 38.7% and 38.7%. Among valid responses, accuracy was 95.5%, 89.7% and 86.2% across the runs, while recall was 94.1%, 84.2% and 81.8%.
A benchmark with a narrow scope
The study reported descriptive percentages, with no confidence intervals or inferential tests. Its controlled testbed did not reproduce the scale, background traffic, diversity or complexity of a real enterprise environment, so the findings are limited to the controlled environment used for the experiment.
The class mix was intentionally enriched for suspicious and anomalous behaviour, and the borderline comparison rested on 20 severity-2 cases and the study-defined threshold. Latency and output validity were measured in the hosted inference setup used by the study, so the experiment does not establish whether the same trade-offs would hold in a larger, more realistic setting.
Paper data and sources
Original title: From Noise to Signal: Improving Security Log Anomaly Detection Using LLMs with Endpoint-Specific Logs
Authors: Christopher Henshaw, Gour Karmakar
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text