An artificial-intelligence system that traces the first wrong move in a team of AI agents reported higher success rates on three benchmark tasks. In tests, DoCtOR's reported improvement over the initial success rate was 22% on HotPotQA, 26% on ChartQAPro and 27% on Mind2Web. The paper also reports higher performance than Reflexion, Retroformer and COPPER.
The manuscript is an arXiv version 1 preprint dated 28 Aug 2026. It presents a diagnose-then-correct workflow that looks for the decisive error, checks an alternative and sends feedback to the relevant agent.
A referee for the team
DoCtOR has two components and a three-stage reflection workflow. Diagnosis identifies the decisive error step and agent, correction generates a corrected step, and a reflector produces targeted reflection for that agent.
Its ProFA module assigns each reasoning step a correctness score from 0 to 1. At a threshold of 0.5, the system converts that score into a correct-or-incorrect marker and defines the decisive error as the earliest step marked incorrect. The aim is to locate where the reasoning first fails, rather than simply flagging the final answer.
The correction module generates a counterfactual action, meaning an alternative to the original action, for the decisive step and scores it with ProFA. A corrected score above the threshold is accepted; a score at or below it leads to another attempt.
What the tests found
The main evaluation randomly sampled 100 tasks from each of HotPotQA, ChartQAPro and Mind2Web. It used Success Rate with F1-based soft answer matching and assessed ProFA with agent-level accuracy and step-level accuracy.
On the attribution data, the authors summarized ProFA as reaching approximately 80% agent-level accuracy on held-in data and about 50% on held-out data. Step-level accuracy was reported as 53%, 78% and 42% in the held-in results, and 38% and 20% in the held-out results. The contribution summary reported improvements over existing methods ranging from 4% to 35% for agent-level accuracy and from 9% to 28% for step-level accuracy.
The attribution model was trained from failure logs built from 1,000 extracted HotPotQA tasks, 500 ChartQAPro tasks and 1,000 Mind2Web tasks. Those collections yielded 667, 379 and 711 failure logs respectively. Initial automated annotation was followed by review from three researchers, and the data were divided into training, validation and testing portions in a 7:1.5:1.5 ratio.
DoCtOR training used 3,565 reflection instances and a three-stage pipeline consisting of supervised fine-tuning, reward modeling and policy optimization, with LoRA used in training. The study also compared the full system with versions that removed individual components. Every tested removal was associated with lower performance across datasets, and the largest reported degradation followed removal of both diagnosis and correction.
A smaller reflection footprint
The authors also tested the paradigm with prompt-based Reflexion. They reported consistent but modest gains for GPT-4o-based Reflexion across all three datasets, and substantial improvements for GPT-3.5-based Reflexion on ChartQAPro and Mind2Web.
A separate test found that using only the steps after the decisive error produced performance comparable to using the complete multi-agent trajectory.
The threshold analysis found that 0.5 generally gave the best reported performance across datasets, while the pattern of improvement remained robust across the thresholds tested.
What the evidence does not establish
The headline figures are benchmark comparisons. The reported percentage improvements are not accompanied in the supplied analysis by exact baseline success rates, confidence intervals or p-values.
The evaluation focused on task-oriented collaborations with clear success criteria, primarily used English-language datasets and involved relatively small multi-agent teams. Open-ended or subjective tasks, multilingual or culturally diverse settings, and larger teams were left untested.
Several supporting findings are also reported without exact values or uncertainty estimates, including the prompt-based reflection comparisons, trajectory-scope test, ablations and threshold analysis. That leaves the size and reliability of those differences harder to judge.
Paper data and sources
Original title: Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration
Authors: Xiaoqing Wang, Keman Huang, Bin Liang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text