On the same 57 questions, a search agent found much less of the relevant evidence when the benchmark was run over a huge, benchmark-independent web corpus, even though it made more search calls. Accuracy was 80.7% on BrowseComp-PlusCM, versus 86.0% on BrowseComp-Plus. Evidence recall—the share of relevant documents that appeared in the search results—was 21.4% versus 84.3%, while search calls were 63% higher on BrowseComp-PlusCM.
The methods preprint asks whether fixing a large independent corpus can make it easier to separate an agent’s contribution from the retrieval system’s contribution. The projected benchmark, BrowseComp-PlusCM, moves BrowseComp-Plus evidence into ClimbMix, a web-text mixture with 553 million documents and 400 billion tokens built without reference to a benchmark.
How the benchmark was rebuilt
The researchers started with 830 BrowseComp-Plus test questions. The projection pipeline broke each question into required or confirmatory atomic hops—individual facts or checks—then searched ClimbMix with BM25, a text-matching retrieval method, and attached documents that stated each hop. Successive filters left 326 questions judged answerable, 65 that passed all-hop verification and 57 after human review.
An independent checking agent was limited to the supplied documents, and the authors manually checked that every required hop had a supporting document. Questions without confident grounding were discarded.
The corpus changed the retrieval picture
In the paired evaluation, the authors kept the 57 questions, retrieval interface, judging setup and no-supplied-document condition the same. Only the underlying BM25 index changed between BrowseComp-Plus and BrowseComp-PlusCM.
The pattern also appeared in two weaker configurations. Gemma 4 31B IT fell from 26.3% to 15.8% in accuracy and from 24.9% to 2.8% in evidence recall when moving to BrowseComp-PlusCM; its average tool calls edged down from 24.5 to 23.4 per question.
Qwen 3.5 9B moved from 14.0% to 12.3% accuracy and from 19.3% to 2.6% recall, while its average calls rose from 33.4 to 37.9.
When relevance judgments were supplied in context, the strongest evaluated agent answered all 57 questions correctly. The authors interpret this as evidence that the identified documents were sufficient once they were available.
A closed-book diagnostic—an answer attempt without retrieval or other tools—showed why accuracy is not a pure measure of search. GPT–5.6 Sol answered 46.1% of all 830 source questions and 70.2% of the projected 57; Gemma and Qwen answered 1.8% and 0.0%. The result means stored knowledge can affect the score alongside retrieval.
The boundaries of the result
The final 57-question set is a heavily filtered subset, not a verdict on all 830 questions. The paper does not show that the questions dropped along the way were unanswerable in ClimbMix; they failed one or more projection criteria. The evaluation also used only a BM25 index, so dense, hybrid and reranked alternatives were not tested.
The relevance judgments carry another caveat. Of 347 question–hop pairs, 40 were supported by at most two documents, and the authors warn that aggregate retrieval scores can be dominated by redundant hops and may not track answer accuracy closely. Evidence recall counts documents shown in search results, not necessarily the documents the agent actually used.
The authors also say the agentic stages are not exactly reproducible: a rerun can produce an equivalent but not identical question set. The released results serve as the canonical record.
They release the benchmark, projection pipeline and per-question pipeline records, but withhold the hop decompositions and per-hop relevance judgments.
Taken together, the results make a benchmark-design point: in this 57-question, human-verified comparison, the choice of corpus was associated with large differences in the amount of evidence agents surfaced, while the strongest agent answered every question when judged documents were supplied. Whether the pattern transfers to other corpora, retrieval systems or benchmarks remains open.
The document is an arXiv v1 preprint dated 20 Aug 2026.
Paper data and sources
Original title: Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search
Authors: Sahel Sharifymoghaddam, Lingwei Gu, Yijun Ge, Jimmy Lin
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text