A preprint studying failures in LLM multi-agent systems reports a 20.15% overall repair rate for a symptom-guided intervention at a selected suspicious node, compared with 6.90% for an unguided full rerun. The comparison used one selective-replay intervention for the node method and up to three complete attempts for the rerun, so the figures describe performance under the study's stated budgets.
The wider question is whether a system has actually repaired a recorded failure or has simply generated a different outcome through language-model resampling. That distinction matters because a full rerun can end differently without showing that the original failure mechanism was corrected.
A rerun can change the answer
In a separate instability check, 54 executions that had initially succeeded were each rerun three times. Of the 162 rerun attempts, 85 failed, and 39 of the 54 original cases regressed at least once.
That pattern is why the study treats a changed terminal result cautiously: a later pass or failure can reflect stochastic variation in execution, rather than a demonstrable correction of the recorded problem. The central test is therefore whether a method can distinguish a reproduced failure from a merely different outcome and use the recorded execution to guide repair.
What selective replay changes
SymTrace Replay reconstructs the recorded execution prefix before a designated target, then resumes live execution from that intervention point. The downstream suffix is generated live, so the method holds the earlier part of the run to the recorded trace while leaving later behavior open to variation.
Across 536 evaluator-confirmed source failures, replay produced higher aggregate same-failure reproduction than an unguided full rerun under both reported settings: 80.78% versus 67.97% at rep1, and 52.43% versus 41.42% at rep3. The replayed prefix nodes also reached 100.00% content-hash exactness, essentially a matching digital fingerprint for the reproduced content, although the live target and suffix still introduced variation and automated evaluation remained a source of uncertainty.
The sample came from a fixed pool of 200 tasks that generated 600 initial task-system executions. The evaluation retained 536 failures and excluded 64 executions accepted by the evaluator; the failures comprised 462 from WebArena-Verified and 74 from AssistantBench, spread across AG2 with 171 executions, CrewAI with 184 and Magentic-One with 181.
The broad repair baselines lagged
At the task level, the aggregate pass@3 results - whether any of up to three attempts passed - were 6.90% for Unguided Full Rerun, 4.29% for Self-Reflection and 3.73% for Critic-Agent. Unguided Full Rerun had the highest rate in each of the three multi-agent systems.
Neither Self-Reflection nor Critic-Agent exceeded the unguided rerun in the aggregate result. Task-level methods were allowed up to three complete attempts, while the node-level method used one intervention; the comparison fixed the task input, multi-agent system, model alias, temperature and native evaluator across matched source failures.
The node-level result was 20.15% overall with one selective-replay intervention, versus 6.90% for Unguided Full Rerun with up to three complete attempts. The reported rates for the three systems were 16.37% for AG2, 25.00% for CrewAI and 18.78% for Magentic-One, with higher rates across all three systems than the unguided method.
The abstract describes the 20.15% repaired-case result as a 191.89% improvement over state-of-the-art repair methods. That relative figure is the reported comparison; the supplied analysis does not provide a confidence interval or calculation for it.
A consistent result with clear boundaries
Within the study's paired analysis, all 15 RQ3 comparisons showed a positive repair-rate difference for Suspicious-Node Intervention. Each 95% confidence interval excluded zero, and all comparisons remained significant after the within-system Holm correction, with pH below 0.05.
The experiments used deepseek-v4-flash at temperature 0.00. Repair proportions were reported with 95% Wilson intervals, rate differences were estimated using 10,000 case-level paired bootstrap resamples, and exact McNemar tests were used for paired yes-or-no outcomes, with Holm correction for multiple comparisons.
An audit checked the reliability of automated language-model judgments. In a stratified sample of 72 judgments, 64 agreed with the audit, five were confirmed disagreements and three were uncertain. Weighted estimates put confirmed disagreement at 3.30%, uncertainty at 3.67% and a combined sensitivity upper bound at 6.96%.
For the underlying case labels, three annotators independently reviewed each case, and a fourth reviewer examined the complete traces, initial annotations and supporting evidence before producing the final annotation.
The result is still a benchmark result
The scope is narrow. The authors caution that benchmark-derived tasks may not capture the diversity of failures in deployed multi-agent systems, and that results from a single model may not generalize to other model families, providers, scales or future versions.
The RQ3 result also combines two choices: selecting a target node from symptoms and supplying repair guidance at that node. Because those pieces were evaluated jointly, the study does not identify their separate contributions.
The document is labeled arXiv:2608.25920v1 [cs.AI] and dated 26 Aug 2026. Its supplement is described as containing the complete SymTrace source code and complete SymFail dataset.
Taken together, the findings support a more demanding definition of repair: a later pass should be judged against the recorded failure, not only against a fresh outcome. That is the preprint's methodological argument; its benchmark results do not establish how well the approach will generalize beyond the tested setting.
Paper data and sources
Original title: Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
Authors: Zhongwen Luan, Xiaoyu Zhang, Ming Hu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text