An AI agent harness that chooses which tasks to use when testing its own proposed changes reported higher held-out scores than competing approaches in a computational evaluation, with average gains of 7.6% to 13.6% across three harnesses and four benchmarks. The result points to the value of targeted verification, but it does not by itself establish that the method caused the improvement.
A test of the harness itself
The study asks whether an agent harness can autonomously and continually evolve from evidence generated during task execution. HarnessLens explores both the task space and user-configurable parts of a harness, uses execution trajectories to propose candidate modifications, and selectively checks those candidates on tasks linked to the behavior that prompted the change. An attributable-evidence gate then determines whether the proposed update has enough support to move forward.
The evaluation covered OpenCode, Codex and Pi across Retail, Banking Knowledge, Terminal-Bench 2.0 and the Challenging subset of BIRD Mini-Dev. For each benchmark, the evolution process used a random sample of 30 TRAIN tasks. TEST tasks were kept separate and inaccessible during evolution, using the official split where available or the remaining tasks. The final outcome was held-out TEST pass@1, the study's one-trial task success measure.
The TRAIN controller used two trials per task. All language-model agent and evolution roles used deepseek-v4-flash-preview; external information retrieval tools were disabled and permission controls were enforced.
The gains were not uniform
HarnessLens achieved the best or tied-best pass rate in eight of the 12 harness-benchmark pairs and never fell below the initial harness, H0. When scores tied, the procedure returned the unchanged harness rather than forcing an update. Across the evaluated harnesses, the authors report average held-out improvements ranging from 7.6% to 13.6%.
The individual scores varied sharply by setting. OpenCode recorded 85.00% on Retail, 25.37% on Banking, 33.90% on Terminal-Bench 2.0 and 45.83% on BIRD, for a 47.53% average. Codex recorded 80.00%, 13.43%, 35.59% and 47.22% on the same benchmarks, for a 44.06% average. Pi recorded 85.00% on Retail and 33.33% on Banking in the reported figures.
The guardrail behind each update
The proposed changes were tested on behavior-relevant tasks rather than only on a broad pool. Each verification batch contained at least five distinct tasks, including a conversion task and one tied to the trajectory that supported the proposal. Current and candidate harnesses were evaluated under matched conditions, with fresh seeds used for confirmation.
Advancement required attributable positive evidence and no attributable regression. An update also had to improve the primary metric in a confirmation batch; preserving the current score was not enough. In practical terms, a change had to show a traceable benefit without a traceable failure before it could replace the existing harness.
An OpenCode ablation reported weaker results when either behavior-aware selection or the attributable gate was removed. Across Retail, Banking and BIRD, HarnessLens scored 85.00%, 25.37% and 45.83%. The initial H0, fixed-batch and random-batch variants each scored 75.00%, 20.90% and 37.50%; the RHO-based batch scored 75.00%, 20.90% and 38.89%, while the metric-only gate scored 80.00%, 20.90% and 37.50%.
A smaller budget, with an accounting caveat
HarnessLens was capped at 200 total interaction units, including task rollouts and language-model sessions. The configured maxima for the comparison methods were 4,800 TRAIN rollouts for Self-Harness, 660 for Meta-Harness and 300 for HarnessFix. Those figures are not a compute-matched price tag: the baselines counted rollouts alone, while HarnessLens counted both trials and sessions.
A decision-level trace shows how the gate could affect the final result. In an illustrative OpenCode-BIRD run, TEST pass@1 rose from 27 of 72 tasks, or 37.50%, to 33 of 72, or 45.83%, an increase of 8.33 percentage points. Re-scoring 19 recorded iterations with a metric-only rule and with the attributable-evidence gate produced different decisions seven times. The metric-only rule would have accepted 10 modifications, compared with five under the gate. The analysis did not build metric-only harnesses or measure their TEST performance, so it cannot show that the metric-only alternative would have scored lower.
What the preprint leaves open
The evidence remains narrow. The evaluation used one model family, three harnesses and four public benchmarks, and broader harness architectures and open-ended deployment environments were not fully validated. All reported comparisons came from this fixed evaluation setting, so the differences should be read as evidence from the study rather than proof of a cause-and-effect advantage.
The TEST results used one fresh trial per task, and no confidence intervals or formal inferential uncertainty estimates were reported. Budget accounting also did not normalize token use, latency or monetary cost across roles and benchmarks. Whether the ranking would hold with repeated TEST trials or cost-normalized budgets remains unresolved.
The work is arXiv preprint 2608.27311v1, dated 27 August 2026, with no journal or peer-review status reported in the supplied metadata. The abstract says code is available through a GitHub repository. Funding was not reported; the document includes an AI-assistance disclosure.
Paper data and sources
Original title: Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
Authors: Jinghan Xu, Yikai Zhang, Aili Chen et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-27
DOI: Not available
Original paper · Full text