Coding agents built most of the repositories they were asked to create from scratch in the released benchmark, but the results also showed how a nearly complete project can still fail a strict final check. GPT-6 Astra paired with Codex solved 10 of 11 tasks, Claude Opus 5.5 with Claude Code solved 9, and Grok 4.7 High with Cursor CLI solved 7. Those results correspond to pass rates of 90.9%, 81.8% and 63.6%.
The benchmark treated success as an all-or-nothing result. An agent received a public product requirements document, an Interface Contract, a reproducible toolchain and an empty workspace. It then had to create the repository, install dependencies, build when needed and submit its work for evaluation. A submission earned a passing reward only when it passed every hidden acceptance test.
That standard matters because the task was not simply to produce code that looked plausible. The evaluation withheld the tests, blocked access to code-hosting sites and upstream implementations, and injected the tests into a fresh judge container. The protocol also required an explicit submission and excluded attempts to repair the code during judging or to have a language model decide whether it was good enough.
A demanding test of repository building
The released benchmark contains 11 tasks drawn from version-pinned open-source repositories. Eight are in Python, with one task each in TypeScript, Go and C++. Across the set, the tasks contain 87 feature points and 87 acceptance modules, with between 4 and 18 modules per task. The combined product requirements and interface specifications range from 10,829 to 33,408 words, with a median length of 19,358 words.
The task designers used each project's source, documentation and tests, then required executable checks, a rebuilt environment and a successful reference implementation before release. They also used adversarial checks intended to reject incorrect implementations. This was designed to test whether an agent could follow a broad specification and deliver a working repository in its native ecosystem.
The comparison covered three model-and-harness configurations, each run once on each of the 11 tasks. Both the solving and judging phases had a 7,200-second limit. The study compared the listed combinations as configurations, rather than holding a model fixed and swapping only the harness.
Most failures were close calls
The seven failed submissions were not broad collapses. They passed between 89.9% and 99.4% of the hidden tests, and all were submitted before the budget ended and judged to completion. None failed because of a timeout, a build error or a missing package.
The pattern of the remaining errors differed by configuration. Grok's failures were concentrated in subsystem failures, which made up 45 tests, or 52% of its recorded failure count. Claude's failures were dominated by long-tail rules, accounting for 12 tests, or 67%. GPT's failures were mostly single omissions, accounting for 13 tests, or 62%, while its analysis recorded no subsystem failures. The distinction between long-tail rules and subsystem problems involved judgment.
For GPT-6 Astra, the analysis assigned 21 failed tests to eight root causes. Three of those causes accounted for 13 tests and were each described as a one-line fix. The finding suggests that, once an agent is close to completing a repository, a small number of overlooked details can determine whether the whole submission passes the benchmark's binary test.
What the result does, and does not, show
The authors interpret this as a shift in the type of failure: the weakest configuration showed more breakdowns in subsystems, while the stronger configurations more often missed long-tail rules or isolated requirements. They point to specification-driven verification and training assets focused on rarely exercised behaviors as possible ways to target those gaps.
The language breakdown is too thin to support claims about which programming language is harder. On the eight Python tasks, the configurations solved 8, 7 and 7 tasks. On the three non-Python tasks combined, they solved 2, 2 and 0. Because TypeScript, Go and C++ each appear only once, those figures are descriptive rather than a language-level comparison.
The study is also limited by its scale and design. It has 11 tasks, only three configurations and one run per task, so it cannot show how much results vary across repeated runs. The ranking is descriptive rather than a definitive test of model or tool superiority, because the release did not compare harnesses for a fixed model. The seed projects are well known, which may produce higher pass rates than genuinely unseen repositories. The authors say an expansion to 30 to 50 tasks is in progress.
The benchmark therefore offers a precise result rather than a broad verdict on coding agents. Under this protocol, agents often produced repositories that passed nearly all hidden checks, but the all-or-nothing score made a small number of missed requirements decisive. The observed failure patterns point to different weak spots across configurations, from subsystems to long-tail rules and isolated omissions.
The evaluation harness, released tasks, hidden suites, environment recipes and experiment scripts are reported as available, while the task-authoring pipeline is not released. The paper is identified in the supplied metadata as an arXiv preprint dated 1 October 2026.
Paper data and sources
Original title: Zero2Repo: Can Coding Agents Build Repositories from Scratch?
Authors: Pei Yang, Tianyu Shi, Yuhang Yao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text