A new arXiv preprint reports a way to compare candidate AI-agent harnesses—the setups being optimized around a fixed language model—without evaluating every candidate on every validation task. Task-CoEvolve selects tasks using earlier outcomes and estimates each candidate’s score across the full task set while accounting for the chance each task was sampled.
In online text-classification experiments, the method reported 49.3% accuracy at a 20% evaluation budget and 47.6% at a 7% budget. Those results were 2.1 and 2.4 percentage points higher, respectively, than the study’s Naive protocol, which used a fixed subset of tasks.
Reported savings with near-full-search results
At the 7% budget, the reported accuracy was 47.6%, compared with 41.6% for the few-shot starting setup, while Task-CoEvolve used 16 times fewer samples than full-set search. The online tests used GPT-OSS-120B at temperature 0 on LawBench, Symptom2Disease and USPTO-50k. They ran for 20 evolution iterations, with three candidate harnesses per iteration, 60 candidates in total and three random-seed runs per setting.
Terminal-Bench 2.1 provided a second benchmark, with 89 tasks. At a 20% budget, Task-CoEvolve’s average pass rate was 51.7%, compared with 52.8% for full search. The model-specific rates were 61.8% for GPT-5.6-Luna and 41.6% for Qwen3.6-35B-A3B.
The reported Terminal-Bench search-cost reductions ranged from 67% to 80%. For GPT-5.6-Luna, input-token use fell from 2,888 million tokens and 22.2 hours under full search to 579 million tokens and 11.5 hours with Task-CoEvolve. For Qwen3.6-35B-A3B, it fell from 741 million tokens and 38.0 hours to 246 million tokens and 20.5 hours.
The method changes which tasks matter most
Task-CoEvolve uses variance-weighted selection based on the historical success rates of individual tasks. It then adjusts its estimate of a candidate’s full-set score for the probability that each task was sampled.
An ablation analysis showed a stepwise rise in the online-classification mean at the 20% budget: 47.2±0.6 for the fixed-subset approach, 48.2±0.5 after random resampling, 48.8±1.4 after full-score estimation and 49.3±0.8 after variance-weighted selection. The figures are reported across three runs for each configuration.
The estimated ranking was much weaker at the 7% budget than at 20%. The rank correlation between estimated and true scores was 0.13 at 7% and 0.62 at 20%. The selected candidate ranked 10th of 60 at 7%, with a true score of 46.4% versus 47.8% for the best candidate; at 20%, it ranked 12th, with 51.3% versus 54.7%.
Important limits on what the results show
Terminal-Bench used the same 89 tasks during the search and for the final pass-rate calculation, so that final figure was not independently held out. The reported results are point estimates without standard deviations or confidence intervals, and infrastructure failures could be retried up to twice, introducing evaluation noise.
The evidence is confined to computational benchmark experiments involving the reported classification datasets, models, task pools and sampling budgets. The online results used three runs per setting, while uncertainty for Terminal-Bench was not reported; whether the method generalizes to other models or task distributions remains unresolved.
The work is an arXiv preprint, version 1 dated 20 August 2026. Its abstract says code will be released on GitHub, but the supplied analysis does not report a current release status.
Paper data and sources
Original title: Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Authors: Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text