Writing a working change across a large software system remains a steep test for coding agents. In LoLBench, the highest observed resolved rate was 14%, the top completed rate was 26.3%, and the best fail-to-pass, or F2P, pass rate was 52.7%. Opus 5 on Claude Code and mini-SWE-agent reached the top resolved rate, while Claude Code with Opus 5 led the completed and F2P measures. F2P tests are meant to fail or error before the reference implementation and pass afterward; regression tests must pass both before and after. The figures come from the configurations evaluated in the benchmark, not from a test of every coding-agent setup.
A test built around large software changes
LoLBench was built to examine both perception, meaning the ability to ground a user's intent and high-level design, and implementation, the production of code changes across the proposal-to-implementation process. It contains 100 tasks across 29 software systems in five domains. Proposals averaged about 5,000 words, the systems about 2.4 million source lines of code, and the implementation pull requests about 5,500 lines of code. That scale makes the benchmark a test of changes that can span a system, rather than isolated code completion.
The proposal and pull-request pairs were retained only under specific criteria. A qualifying system had at least 50,000 source lines of code; the proposal needed an implementable section; and the pull request had to change at least 1,000 lines across at least 10 files, with more than half of changed lines in source code. The proposal had to come before and match exactly one merged pull request, and tests had to check the proposed behavior.
The main evaluation paired six models with five agent scaffolds. Provider issues left 28 agents in the reported run. Models used the second-highest reasoning effort. Each attempt had a six-hour wall-clock limit and ended after one hour without progress; apart from retries, each agent received one attempt per task.
The search for relevant code dominated
Once agents began working, finding the right places in the code dominated their activity. Code localization, the search for relevant files and functions, was the largest activity phase for 26 of 28 agents during initial solution generation and for 21 during repair. It accounted for 60.4% of turns in initial generation and 44.2% in repair.
The coverage measures show how incomplete that search remained. Inspection recall, the proportion of relevant code found, ranged from 46.4% to 86.4% for files and from 31.9% to 83.1% for functions. Inspection precision, the proportion of inspected items that were relevant, ranged from 16.1% to 36.3% for files and from 6.8% to 19.2% for functions. Modification recall remained between 36.9% and 65.0% for files and between 20.8% and 53.4% for functions.
The failure analysis pointed in the same direction. Code localization had the largest failure attribution, at 30.9 units, followed by code editing at 25.3; missing cross-module context accounted for an average of 30.0 localization units. These are fractional diagnostic weights, not percentages of tasks. The analysis combined predefined rules for observable localization, editing, verification, repair and tool-use signals with an outcome-blinded LLM judge for semantic understanding and planning. Multi-mode solution failures were normalized so each contributed one attribution unit.
Performance also varied more across models than across scaffolds, the surrounding agent setup. Relative standard deviation, a measure of relative spread, averaged 62.0% across models versus 24.0% across scaffolds for completed rate. For F2P pass rate, the corresponding figures were 44.5% and 17.3%. Across those comparisons, model variability was about 2.6 times larger.
More context helped, with important limits
Extra implementation detail was associated with better results in a separate paired comparison. Three agents were compared with proposal-only input and with a specification supplemented by a reference-derived file tree and API specification. The specification condition had resolved-rate differences of 16 to 22 percentage points, completed-rate differences of 13.7 to 19.2 points, and F2P differences of 13.3 to 27.6 points. For Claude Code with Opus 5, the F2P pass rate was 52.7% with the baseline input and 72.7% with the specification.
That comparison does not show that the added guidance would produce the same gains in general. It was nonrandomized, covered only three selected agents, and used information derived from the reference implementation. The paper reports descriptive comparisons and does not provide confidence intervals for these results.
The benchmark also measures only one part of software quality. Its fail-to-pass and pass-to-pass checks assess functional correctness, but they do not separately assess readability, maintainability or consistency with project design conventions. The evaluation covered 28 agents built from six models and five scaffolds, generally with one attempt per task, so other agents and settings, as well as run-to-run variability, were not fully characterized.
The document is an arXiv preprint, version 1 dated 29 Sep 2026. The paper states that the LoLBench dataset is available at https://huggingface.co/datasets/lolbench26/LoLBench. Taken together, the findings identify cross-module understanding as a major bottleneck in these tasks, while showing that more explicit structural guidance was linked to better results in a small paired test.
Paper data and sources
Original title: LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems
Authors: Yun Peng, Zihan Wu, Zeyang Zhuang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text