An arXiv preprint dated 20 Aug 2026 tested whether AI coding agents could improve training algorithms inside research repositories. In AI4AI-Bench, six systems were run in 29 configurations across 10 tasks each, producing 290 evaluation cells. The mean mapped score was 0.166. The strongest system averaged 0.250, while the best configuration—Claude Opus 5 at medium effort—averaged 0.288.
The main pattern was a split in what the agents chose to edit. A separate analysis found that 17 of 280 submissions had no classifiable change. Among the remaining 263, 141 stayed entirely on the run side—changing how a run was set up or executed—while 122 touched the learning procedure itself.
That distinction matters because the benchmark was built to isolate algorithmic design. It covers 10 repositories, each representing a distinct training-algorithm family. In practical terms, a learning-side change edits how the model learns; a run-side-only change leaves that procedure untouched.
Scores were mapped separately for each task against three anchors: 0 for an uninformative model, 0.1 for the shipped repository algorithm and 1.0 for the task optimum. On that scale, 124 of the 290 cells fell below 0.1. The overall average of 0.166 was above the shipped-algorithm anchor but remained below the 1.0 optimum anchor.
The edits that reached learning code scored higher
When the researchers compared the two kinds of edit, submissions that touched the learning procedure averaged 0.226, versus 0.126 for run-side-only submissions. The difference was 0.100, with a reported standard error of 0.022.
That gap should not be read as proof that learning-side edits caused higher scores. The comparison was not randomized, and the authors explicitly note that stronger systems were more likely to reach the learning procedure. The score difference may therefore reflect both the systems making the changes and the changes themselves.
Reasoning effort was associated with more learning-side edits and higher mean scores. From the lowest to the highest effort level, the share of submissions touching the learning algorithm rose from 8.0% to 64.0%. Mean score rose from 0.094 to 0.196, a difference of 0.102. Within Codex, the mean rose from 0.094 at the lowest effort to 0.204 at the highest.
Within Codex, higher effort also came with a larger exploration footprint. The median configuration went from 4 to 16 evaluations and from 18 to 246 edited lines. Output rose from 11,000 to 109,000 tokens, while median exploration cost rose from $1.69 to $34.60 per task.
Some attempts failed before producing a useful benchmark result. Nineteen cells scored zero: eight ended without a usable patch, and 11 submitted a complete patch but produced no satisfying formal artifact. All 19 terminated normally.
A controlled test with narrow limits
Repositories were admitted using three criteria: an actual shipped algorithm, a frozen starting model and a reproducible metric runnable within a half-day on one B300 GPU. The benchmark therefore began with existing research code and a defined starting point for each task.
Each agent received four hours to explore on one B300 GPU, with a fast proxy metric used during exploration. Submitted code was replayed from a clean start for up to 12 hours. The unmodified repository algorithm was run under the same procedure, hardware, budget, evaluator and evaluation asset, leaving source code as the controlled difference.
Those controls make the baseline comparison more consistent, but they do not turn the learning-side groups into randomized groups. Because stronger systems were more likely to reach the learning procedure, the learning-side average is best treated as an observed association within this benchmark.
The scope is correspondingly narrow. The evidence covers 10 repositories, six systems, 29 configurations and the reported clean-start replays. It measures what agents changed in those codebases under those budgets; it does not by itself show that an improved training algorithm produced a better successor agent.
The authors say they release the task suite, evaluators and every scored submission to support repeatable measurement. The supplied document is identified as arXiv version 1, dated 20 Aug 2026.
Taken together, the benchmark points to a gap between editing a research codebase and redesigning the procedure that drives learning. Learning-side submissions scored higher on average, yet run-side-only submissions remained more common among classifiable submissions. The authors interpret the wider pattern as current agents more often recovering competent defaults than designing beyond them.
Paper data and sources
Original title: AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
Authors: Yizhe Chi, Wenyi Li, Deyao Hong et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text