A framework for collecting multi-agent AI trajectories recorded a dramatic difference in feedback in a direct comparison of original and changed inputs. Across 1,197 same-task pairs, feedback appeared in 99.75% of the perturbed runs, compared with 0.33% in the original-input condition, a 99.42-percentage-point difference. The reported 95% confidence interval, a statistical uncertainty range, ran from 98.91 to 99.83 points.
The preprint's framework, CoEvoke, uses two collection controls: perturbations intended to draw out checking and repair, and restricted reference-based guidance intended to direct investigation. Its corpus contains 4,567 reconstructable trajectories covering 4,130 tasks.
A sharp change in feedback
A second paired comparison covered 1,090 same-task cases, comparing strong-first generation with a weak-to-strong sequence. Feedback was 5.14% in the strong-first condition and 70.46% in the weak-to-strong condition, a 65.32-point difference. The 95% confidence interval for that difference was 62.57 to 68.07 points.
Both conditions solved all 1,197 input-perturbation tasks. But only 732 perturbed runs, or 61.15%, produced a selected correct revision. Another 465 runs, or 38.85%, also passed but did not meet that combined condition. The distinction matters: selected correct revision is a process measure, not the same as overall repair success.
The gains varied by model
Restricted guidance also showed a smaller, model-dependent edge in a fixed software-engineering test. Across the same 80 SWE-smith tasks and five models, the reference-based condition recorded 361 successes out of 400 runs, or 90.25%, compared with 348 out of 400, or 87.00%, for the public-only control. That is a 3.25-point gap. Aggregate net provider-token use was 8.55% lower with reference-based guidance.
The size of the difference was not consistent across models. Reference-informed reasoning differences ranged from 0.85 to 3.75 percentage points, while SWE token reductions ranged from 3.91% to 17.95%. The token totals are descriptive: they do not use a common tokenizer or translate directly into a common monetary cost.
On external workflow benchmarks, the CoEvoke setup had higher success than budget-matched single-agent controls in eight of nine external model-benchmark combinations. The evaluations used unperturbed inputs, public execution evidence and no private-reference guidance, while solving weights and output protocols were held fixed. All three models gained on SWE-bench Verified and ACPBench-Hard. On MathArena, Qwen gained 6.25 points, Gemma gained 3.47 points, and Ministral lost 2.78 points.
Training tests extended the comparison
In a supervised fine-tuning comparison, Perturb-SFT solved 438 of 1,040 ACPBench-Hard tasks, or 42.12%, while Natural-SFT solved 368, or 35.38%, a 6.73-point difference. The comparison drew on 884 shared tasks, 891 decision records and 379 updates.
A separate verifier test on SWE-bench Verified reported higher success in both verifier interfaces. Patch-based verifier training had 32.40% success versus 24.80% for its comparison, while trajectory-based training had 32.80% versus 23.40%.
The evidence remains limited
The quality audit was favorable but limited in scope. It used 240 ratings over 160 trajectories for calibration, then a weighted probability sample of 500 trajectories. In that weighted sample, 93.1% were graded A for interaction quality, 2.0% carried the paper's L1 disclosure label, and no L2 disclosure was detected. Because it was a weighted probability sample rather than a census, the figures are estimates of the corpus.
The paper validates CoEvoke only on small multi-agent teams and limits training to supervised fine-tuning. Whether it works in larger systems, after post-SFT reinforcement learning, or for broader agentic capabilities remains open.
The preprint reports AI assistance but no funding source or conflicts of interest.
Paper data and sources
Original title: CoEvoke: Evoking Multi-Agent Collaboration for Trajectory Collection
Authors: Lin Wu, Shu Yao, Yuheng Wang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text